Oh boy. And here I was thinking about starting another X3 run soon anyway. Perfect timing.
Have you played any sort of space sim game? X-Wing, Freelancer, Freespace, anything of the sort? Those games are much more focused in scope compared to the X series, so it might be worth it to play one or more of them before jumping into the X series. In fact, they're worth playing regardless. I personally recommend Freespace 2
So, right off I'll tell you that I've only ever played the X3 games. I haven't played X1 or X2. Couldn't tell you much about them other than "they're much older and are a bit different in scope and gameplay compared to the X3 games from what I understand.
As for what you should start with, well... I know that while there's an overarching storyline in the games of some sort, none of it is necessary for enjoyment of any individual game. I jumped right into X3 and haven't found myself wanting for story details. So personally, I think you can just jump into X3 if you want the most recent in the main line of X games (because Rebirth was nothing but a rumor and you'll never convince me otherwise).
X3 has three games in its line: Reunion, Terran Conflict, and Albion Prelude. They're all on the same engine and share most of the same base features, but AP is the most recent one and, as such, the one with the least bugs and the most additional features added. Regardless, they're big games, and not particularly focused. They are more based on the idea of letting the player do as they please. Sure there's an ongoing story in each game, but it's entirely optional and is probably more of an overall gameplay tutorial than the meat of the game in itself. I've never actually completed any of the story plots in any of the games; I get too distracted by all the other things I want to do.
Do I like the games? Absolutely. I definitely recommend them. But they're damn complex to start learning at the start, even without considering modding. And you're going to want to mod your game eventually. Vanilla X3 is barebones compared to the features available with mods. That's why, if you've never played a space sim, I recommend a more focused game to start with, like Freespace, which is just a mission-based combat game. But nothing's stopping you from jumping into X3.
It's hard to really get more specific about the game and what you might like or dislike about it without knowing what you're looking for in a game. What your expectations are. How willing you are to learn as you go. Things like that. But ask me anything about the X3 games if you want! I'll answer as best I can.