There's nothing to be nervous about. You should have no problem getting a taxi, but the subway is cheaper, easier, much quicker, and very safe. The subway entrance for the #2 is a block away, and it'll get you to Penn Station in about 15 minutes. Just be sure to cross to the other side of Lenox Avenue for the Downtown #2. (The #3 stops there too, but doesn't always stop at 34th Street/Penn Station, so it's easier to just get the #2.) If you're unsure, just ask any one of the hundreds of people on the street!
It sounds like you haven't been to that neighborhood. Why not go early and explore? The Studio Museum is right nearby and frequently has wonderful exhibits. And there are some excellent restaurants nearby. Before you rush back to Penn Station, stop by Red Rooster, or Sylvia's.