Internet2 Network Upgrade

It's been fun....

2007-10-09T19:37:00.000Z

As some of you already know, the Internet2 transition to Level3 has been complete for the past few weeks. The first clue was probably all the remaining Abilene nodes disappearing from the weathermap last week. They've been shut down and the first is coming out tomorrow. The other nodes will be removed over the next few weeks.

With everything complete, these little updates will cease. This was a wonderful experiment into the benefits (and perils) of direct communication between the Internet2 NOC engineering staff and the community. We learned a bit and may do more in the future in a more ongoing operational capacity, but don't look for that to happen anytime soon.

There are really too many people behind the scenes to thank. I'd like to acknowledge several individuals in particular that spent many sleepless nights on the road helping to make this transition a success. These are the true heroes of the upgrade that really made this all come together.

Hans Addleman
Andrew Lee
Caroline Carver
Jay Duncan
Tom Johnson
Aaron Pagel
AJ Ragusa

In addition the following people found other ways to assist. My thanks go out to them as well:

Steve Peck and Nathan Lucas for keeping the NOC service desk in the loop and assisting with procedural work
Matt Davy for pinch hitting for me when I was out in late March and keeping the CPS network on track throughout the upgrade
Caren Litvanyi for wading through countless spreadsheets and analyzing the minutiae of the suite buildouts
Luke Fowler and Ed Balas for their support on the database and documentation systems
Chris Small for keeping the observatory in step
John Graham for managing the entire Ciena network and alleviating hundreds of hours of concern
Jon-Paul Herron for keeping everyone's sanity at bay

This truly was a team effort and I'm proud to work with such high caliber individuals. If you see them in the hall at a conference, give them a pat on the back. They all deserve it.

So long....

When one problem exposes another

2007-09-14T05:41:00.001Z

Sometimes the network Gods have a funny way of toying with your mind. We noticed some L3 incompletes on the new CalREN circuit yesterday, and we pointed the finger at the optical levels on the CalREN interface. I worked with Level3 early this morning to clean the fiber path between I2 and CalREN. They found a bad port in the path at 600 Wilshire that was causing us to launch -14.3dB toward CalREN- just a hair under the tolerance specs of their card. After we cleaned that up, the L3 incompletes persisted. Turns out it was BPDUs that hadn't been filtered all along. Had they been filtered, we may not have caught the light level problem.

Someone was trying to tell us something. :-)

Anyway, CalREN is clean as a whistle now and should be shifting traffic tonight. I'm going to bed.

CalREN almost migrated in Los Angeles

2007-09-13T13:05:00.000Z

CalREN engineers shifted their traffic over to the new Internet2 link, and noticed a very small trickle of errors on the port. Light levels are showing up right below the threshold, so we'll need to do a bit of cleaning there. Look for that to happen soon.

NGIX-W/NREN migrated to Salt Lake City

2007-09-13T03:20:00.000Z

I migrated the Sunnyvale NGIX-W and NASA connection over to the Salt Lake City router tonight. Always a bit tense when you move that many peers, but this went as smooth as silk. A big thanks to our friends at ESNET who provided a concurrent GigE interface to the Ciena in the Sunnyvale POP so that we could backhaul the circuit to Salt Lake City.

HOPI IP migrated in Los Angeles

2007-09-11T01:26:00.000Z

We were able to get the HOPI 10GigE interface up in Los Angeles today, across our metro fiber between 600 Wilshire and 818 W. 7th. We tried to get the PAIX 10GigE up to a loop in Sunnyvale, but we're having some metro problems in LA. The signal just ain't getting between the buildings. We have a ticket open to troubleshoot that in the AM.

LEARN migrated UT Austin behind them on Friday night, so they're now shut down on Houston. That leaves three BGP sessions left on Houston.

I need to schedule CENIC on LA- the last BGP session on the old Los Angeles router.

After that, it's just PAIX, NGIX-W and Oregon in Sunnyvale. I'll probably move the NGIX/Ames VLANs this Wednesday, along with PAIX if I can get them up tomorrow. Oregon may be a few more days out than that.

Pacific Wave, CENIC and LEARN and all the rest...

2007-09-06T03:21:00.000Z

It's been a month or so. Let me see if I can catch everyone up. We've been mainly busying ourselves with getting some fiber patches and circuits lit on the west coast.

I just migrated all of the Pacific Wave Los Angeles peerings over to the new Los Angeles router. Once we got the link up, it was a painless hot-cut.

We did some metro fiber troubleshooting in Los Angeles this evening and the CENIC/HPR connection is up to the new router. We'll need to coordinate a time to swing their peering over to the new circuit.

LEARN a few weeks ago, and the Texas schools are all working on aggregating behind them. Our piece is done.

USF also looks to be up in Atlanta, so we'll probably be able to shut the old M5 down pretty soon.

We're close on Oregon and PAIX. I'm hoping to get all our pieces worked out this week. There are some interfaces being shuffled around in LA to get them from the old router to the new one. Those should be in place by Friday. We very well could have everyone off the old routers by the middle of next week, if things click. Probably more realistic to expect two more weeks, though.

Houston T640/Ciena installed

2007-07-26T12:53:00.000Z

The Houston T640 was installed in Houston yesterday and the backbone circuits were lit through the site. Our engineers still have some other work to do today to get the Ciena and PCs installed. Overall, things appear to have gone fairly smoothly.

Denver Abilene router shut down

2007-07-12T02:57:00.000Z

Denver was one of my favorite routers. Mainly because I love Boulder so much and always got to spend some time out that way. I feel a bit like the delivery doctor and the executioner on this one since I helped install it just under four years ago with Caroline. Debbie Montano stole a few moments away from the Qwest corporate offices to see how things were going. Unfortunately, we gave her a camera and posed for a few photos.

The sad thing is that I have too many pictures of me standing next to network equipment with that same cheesy grin on my face, like I just caught a big bass and am holding it up for the camera. At least Caroline looks like she's putting away some tools or something.

Anyway, Denver hasn't been doing much for anyone lately. All networks were moved off by last week and the backbone links weren't in the path for much of anything. It's been turned down, so our engineers can (gingerly) move it to Houston. We need the Denver power controllers in Houston, so we're going to allow Level3 next week to get them installed for us.

Ah, Past Chris. If Future Chris could tell you anything, I'd probably make sure you know how to lace better. And that you should never look away from your hands to pose for a photograph when you're that close to live DC power. Sigh

Los Angeles T640 installed

2007-07-05T01:09:00.000Z

Tom and Hans were able to get the T640 installed in Los Angeles fairly quickly. The link to Kansas City is up to provide IP connectivity while we wait for the high speed IP drop from Level3. The OC-192 to Seattle didn't come up, but that's likely because we have a fiber backwards in the path. We didn't have light to the Westin building when they were running the fibers in the fiber meet me room, so there was no way to judge the polarity of the cable.

I'm taking the next few days off, but Andrew is going to work on getting that circuit up while I'm out.

Denver is coming out next week. Turning it off on Wednesday night....

Los Angeles! Albuquerque! Denver! The updates! The updates!

2007-07-02T14:33:00.000Z

I've been getting many a query from you Internet2 upgrade blog junkies asking whether I was dead or had just turned my back on this whole ballyhoo. Fortunately, neither. I'm very much alive and things have very much been moving forward. I had anticipated a bit of a lull over the last few weeks, but I was (sadly) mistaken. Just a rundown of the big things:

Los Angeles
Savvy I2 blog readers will recall that the Los Angeles router was slated to be installed two weeks ago. We pushed that out a week to accomodate for some power controller installations. We really hate doing DC power from the house since we don't control both ends. So, we went out last week. But there was one, glaring problem: no router. There were a whole host of scheduling mishaps between Fedex and Qwest in Kansas City that meant our router didn't get shipped the prior week. We finally got Fedex on-site on Wednesday to ship it overnight to Los Angeles, but fate decided to stick her long ugly nose into things. There were actually two crates at the Kansas City POP. One contained the router that was going to Los Angeles using Fedex Express Freight. (Note the italicizing of Express Freight. Very important.) The other contained the reclaimed flotsam from the Abilene rack that was going Fedex Freight back to Bloomington. Unfortunately, both got picked up by regular Freight, which meant no guaranteed delivery times in Los Angeles. Up until the very end, it looked like things were going to work out, but it wasn't meant to be. Fate 1, Upgrade Team 0. So, we're sending two engineers back out tomorrow to finish the install and commission the node. They're going to miss all the area fireworks, so they deserve a round of applause

Albuquerque
Though the Ciena was installed in Albuquerque in early June, the phone line proved to be more troublesome than we thought. The Qwest phone group is totally separate than the Qwest transport group, meaning that it was essentially two different companies talking to each other. That slowed things down to the typical speed we'd see with most phone line installs. Add the fact that the Albuquerque POP is essentially a bundle of disjoint buildings in the desert, and you have a recipe for slowness. Fortunately, Tom Johnson stayed on top of things and we got the phone line delivered last Tuesday. The cross-connect into our equipment was completed on Friday. In the interim, ESNET graciously helped us out with a Layer2 VPN to the Washington DC POP so we could commission the node. (Recall that we don't turn the Ciena nodes up. Ciena does.) That paved the way to get the University of New Mexico connected and working up to Salt Lake City.

Denver
Aside from UNM, there were a few others on the Denver router, which we plan to pull out next week. Both Front Range Gigapop and the Utah Education Network are up on the new router and passing traffic. There was a bit of scare when the fiber cross-connect in SLC wouldn't allow us to ping the remote ends. Turns out it was a hardware problem with our router card. Got it replaced and everything looks spiffy

***

So what's next for our intrepid crew? Not too much left, really- in terms of physical work. We're yanking Denver next week. It's being sent to Houston where it will be installed two weeks later (to allow time for the power controllers to be installed). After that, there's a pending Ciena install in Seattle, but we don't have a time frame nailed down for that. There's obviously a lot of work left to transition customers to the new network in LA and Houston, but most of that is something that can be done remotely. Of course, we have to pull out the Sunnyvale, LA and Houston routers at some point in the next few months. I'm working on solidifying that schedule right now.

Kansas City router begins journey to LA today

2007-06-14T12:35:00.000Z

Our faithful T640 in Kanasas City (Qwest) will begin its magical journey to Los Angeles today as Jay and Andrew yank it out. I turned the chassis down last night. For some reason, the data collectors that are polling the Abilene routers have tanked, so that explains the state of the weathermap. I'm assured that it will be fixed soon to reflect the current topology.

Level3 by the numbers

2007-06-11T15:03:00.001Z

With the Level3 build of the Internet2 network largely completed, they shared some fun stats with us:

13,500 Long Haul route miles
Deployed and configured over 300 Infinera Network Elements
Day 1 capacity of 100Gbs
Built 27 custom Colocation Suites representing 3,365 SQFT of space including:

94 Racks

60 Individual bulk cables with 48 & 96 fiber count

Deployed 64 metro fiber route miles
Enabled connectivity from Houston-to-El Paso & El Paso-to-Denver for L(3) to Internet2/GRNOC and ESnet NOC access
Developed the Virtual Network Operations Center – Provisioning and Troubleshooting Dashboard
Product Summary: Wavelengths, HSIP, Private Lines, Metro Fiber, Colocation, Cross-Connects, Power Controllers, and GPS timing

A job well done by Level3!

But now's not the time to rest on our laurels. We still have work to do on the IP network migration and Ciena network turnup. Kansas City turndown this week, and Los Angeles install next week. Does the fun ever stop? :-)

How sweet it is!

2007-06-07T23:14:00.000Z

This is the week of good news.

A bit of recap and a few other bits of news:

SLC to Seattle OC-192 up across metro fiber
Level3 finished the DWS to Los Angeles
Albuquerque and Sunnyvale install done, meaning all Ciena-only sites have been deployed
the University of Memphis circuit is up and ready for route exchange
Kansas City can come out next week
LA suite delivered a bit ahead of schedule and will be accepted on Tuesday

The crew in Albuquerque deserve an extra round of applause for working on 3 hours of sleep and handling things so well in one of the most punishing of sites.

early travel logistics problems in Albuquerque

2007-06-07T15:14:00.000Z

Our engineers in Albuquerque today decided to jump start the fun by scheduling themselves onto a flight that got cancelled. Jay just sent an update that I'll let speak for itself:

"Landed in Denver, which was bumpy but fine. Found that the flight to Albuquerque cancelled due to wind and the next available flight to Albu, 2:46pm Thursday. Time to Albu driving, according to airline [gentleman], was 4.5 hours. So we rent a car. And begin waiting for our bags to be pulled off the plane. 3.5 hours later, we're told that no bags were forthcoming. Awesome. We hit the road, and find that it's more like 6.5 hours. Bad research. Arrive at hotel 4am local.

Okay, so we're here, but our bags aren't, which means we have no boots, and no hard hats. "

Internet2: From Sea to Shining Sea

2007-06-06T19:20:00.000Z

Yeah, so Internet2 made my 3 month old daughter cry this afternoon.

More to the point, I made my 3 month old daughter cry when I leapt for joy at seeing the Salt Lake City to Seattle OC-192 come up. I still blame the OC-192.

We've been fighting with this one for the past week or so, getting jumpers run and the appropriate patches in the fiber meet me room at the Westin Building. In the end, all worked out well and Internet2 has SONET connectivity from the burroughs of New York to the shipping yards of Seattle.

I hope to get things configured very shortly to begin passing traffic through the new router in Salt Lake City. Now that it's dual homed, we can start moving folks over.

With that, the last two Ciena-only nodes, and the University of Memphis OC-48 in play this week, it could turn out to be a very nice week for the project.

Denver and Baton Rouge Cienas installed

2007-06-05T12:12:00.000Z

Our engineers installed the Baton Rouge and Denver Cienas late last week. Had a bit of trouble in Denver with our out of band router being dead on arrival, so I'll get a replacement out this week. Otherwise, things went well.

This week is Albuquerque and Sunnyvale.

Did you miss it? Salt Lake City installed last week

2007-05-30T22:19:00.001Z

Apparently our network elves have been hard at work in the wee hours of the morning cobbling shoes and installing router nodes. Unbeknownst to many, the Internet2 network peeked its head into the Beehive State and decided to set up shop. It's safely tethered to the Kansas City router now, but we're aiming to change that in the next week with a second lifeline to Seattle. That will make it a full-fledged grown-up router, able to route with the best of them.

Steve Corbato dropped by on Friday just to make sure our boys hadn't electrocuted themselves and posted some photos of the visit. I know what you're going to ask, and the answer is: "No, the POP doesn't have a brew-pub in one of the suites."

For some reason the router still thinks its in Atlanta. :-) I''ll be configuring it tomorrow so it can play in all the reindeer games with the rest of the big boys.

We have two teams heading out West and South to install Denver and Baton Rouge Ciena nodes. We wish them well.

Kansas City backbone and GPN transitioned

2007-05-18T20:47:00.001Z

I'm happy to announce that the Kansas City-Chicago and Kansas City-Atlanta OC-192s are fully in production on the new network and are routing traffic between the three cities. In addition, we now have a new interconnect between the old Abilene network and the new Internet2 network in Kansas City.

Concurrent with that, GPN migrated their 10GigE over to the Level3 POP.

We expect to have the Salt Lake City T640 up next week with connectivity back to Kansas City and (possibly) to Seattle.

Have a good weekend.

KANS T640, GPN transition, El Paso, Atlanta

2007-05-17T22:09:00.000Z

Rest assured, we've been diligently working this week: on not-so-public/exciting/sexy install packets, shipping and documentation. It's all been prep for today, when four major tasks hit.

The Kansas City T640 is fully configured and has backbone links to Atlanta and Chicago. It's been brought into the iBGP mesh and I'm putting the finishing touches on the monitoring/documentation. The Chicago circuit came up yesterday, and the Houston circuit (actually two circuits glued together in Houston in anticipation of the eventual Houston T640 install) was turned up 10 minutes ago.

Second: we nailed down the logistics of the GPN transition to the new network tomorrow. GPN will be moving their router from Qwest to Level3, thus abandoning some dark fiber between the two POPs. Being the opportunistic people that we are, we're both using that fiber to maintain some connectivity into the Qwest POP using the same CWDM splitters we used in Atlanta. The clock starts ticking at 9AM tomorrow on the move. Our side is fairly straightforward, and I'm sending good thoughts toward the local engineers that have to lug the 7600 between POPs.

Third: The El Paso Ciena install is underway. All equipment is on site and I heard good things from the field on the progress made. That should wrap up tonight or early tomorrow morning.

Fourth: The Atlanta T640 was pulled out of the Qwest space and will shortly begin its wonderful journey across the country to Salt Lake City for install next week.

I love it when a plan comes together.

Kansas City Install almost complete

2007-05-11T12:44:00.000Z

With the exception of a few power runs and a bit of documentation, the Kansas City install is largely complete at the Level3 POP. The T640 arrived yesterday and was powered up. We'll be activating the backbone circuit to Chicago today. The circuit to Atlanta (temporarily bypassing Houston until it gets installed) will have to wait for Monday when we slot the OC-192 I had yanked from New York on Thursday).

We're solidifying the details, but there will be an interconnect between the old Abilene router and the new Internet2 Network router via some University of Nebraska metro fiber. We'll be putting another one of those CWDM optical combiner/separators on the link to share the path with GPN. Not sure on the timing of that yet, but it's starting to look like we'll make that happen next Friday. Once that's complete, we'll drop the Qwest OC-192 between the Abilene Kansas City router and the Internet2 router in Chicago. That frees up some much-needed OC-192 PICs and allows us to move forward in Salt Lake City in two weeks' time.

DC router removal saga

2007-05-09T00:55:00.000Z

You may be wondering why I didn't gush over the perfect router removal we had on Monday. That's because it wasn't meant to be. Our engineers' flight to DC got cancelled on Monday morning and they couldn't get rebooked until later in the afternoon. So, they rented a car, put on some tunes and did a Tommy-Boy road trip out to DC. Knowing Jay, there was probably a lot of Rush involved.

Jay and AJ got to the Qwest POP this morning only to find that the empty T640 crate had been "obliterated" in shipping. This is where the driving came in handy. They had a smaller T320 crate that they were able to use for the T640, and the put the rest of the de'commed equipment in their Taurus. Sadly, no more room for Rush.

Unfortunately, the work at Qwest took them pass their badging window at Level3, so they need to stick around another day so they can complete some PC install work early tomorrow morning. Then it's back home to Indiana.

We're attempting to secure another T640 shipping crate from Juniper, but if anyone has one that they've been using as a doghouse or Tornado shelter, drop me a line. I'm off to check Craigslist.

Atlanta and Washington DC Abilene T640s turned down

2007-05-07T21:42:00.000Z

I turned down both the Atlanta and Washington DC T640s just before 1PM EDT this afternoon. This officially removes them from the network, though they stopped routing traffic last week when we moved the Atlanta-Houston circuit over to the new router in Atlanta.

Washington DC is being pulled out tomorrow. It was originally slated for today, but our engineers' flight to DC was cancelled and they won't be able to get in before 5PM this afternoon. That router will make a quick trip to Kansas City for install on Wednesday.

Jacksonville Ciena installed

2007-05-04T19:32:00.000Z

Just a quick note that I got a call from Tom Johnson to tell me that they had finished the Ciena install in Jacksonville.

Now I'll stop posting on Atlanta

2007-05-04T02:50:00.000Z

And now, drumroll, please....

Atlanta.
Is.
Migrated.

Yeah, I can hardly believe it. The past few days have been up and down (mostly down). Yesterday, I'm ashamed to say, I forgot that the GigE port on the M5 that we were swinging the SOX dark fiber to was LC. Of course, the SOX card has SC connectors and the Qwest techs couldn't locate any SC-SC barrel connectors. So, we had to move back the migration to this evening, while we overnighted an SC-LC jumper and connectors to Atlanta.

Trouble is, we got our wires cross and accidentally sent it to the Level3 POP in Atlanta. We were able to locate a local courier that could drive it between the two POPS, so that turned out to be a non-issue, but one that had me on pins and needles for a few hours.

The next problem was port the fiber was plugged into in the Level3 POP. When Jeff was in Atlanta two weeks ago, I had forgotten that we slotted the LH SFP into port 1 of the 4-port GigE card instead of port 0, like I had planned and wrote in the interface description. When I devised the instructions to Level3 for running the fiber on that side on Monday, I put the wrong port down and completely missed it. Fortunately, while away from the office today at 3, the thought magically popped into my head and I had a coworker double check my work. Turns out, I was indeed mistaken, and we had a short window to get Level3 out to the POP. It was 4:30 and we were still waiting on both techs to call in. Fortunately they did...at the same time.

Magically, the connection came up. I say magically, because at this point, I'd probably believe you if you were to tell me there was imps running around in our fibers just trying to keep Atlanta as-is.

All that was left was to migrate USF down to the M5 at 10PM and do the backbone move at 11PM. The USF move went well, though there was some sort of problem with the adjacent PIC to the University of Mississippi. For some reason, all the BGP sessions through that ATM card went down and IP reachability was nil. None of this work should have affected them, but I reset the card anyway. That fixed it for some reason. Hmmm. Need to check that out in the AM.

The backbone move went as smooth as butter. Down on one router, back up on the other within 2 minutes. Even the Qwest techs were amazed at how easy it was.

So, this feels a bit like an Oscar speech, but we really have to extend our thanks to a lot of people. Scott Friedrich at SOX, and all their work really made the difference. Qwest did an awesome job of testing the circuit in advance and working around the multiple changes in our schedule. Level3 was spot-on with the on-demand remote hands and help today with the courier. This was a real team effort and one of those war stories that will linger in my mind forever.

Onward to Kansas City!

Last night's fiber cut

2007-05-02T14:16:00.000Z

Now that the dust has settled on last night's bridge fire, we'd like to extend our gratitude to Level3 for making such a rapid repair. We've dealt with bridge/tunnel/road/railway fires before, and this was probably one of the quickest repair times we've seen for such a disastrous event. This also highlights the flexibility we have to route traffic around such events. Kudos to all involved.