Cisco IOS hints and tricks

Who needs IPv6?

2009-11-20T07:29:00.001+01:00

One of the most common questions asked by our enterprise customers is “Who needs IPv6?” Since IPv6 does not add any significant new functionality (apart from larger address space), you can’t gain much by deploying it in an enterprise network … unless you’re huge enough that the private IPv4 address space (RFC 1918) becomes too confining for you. A good case study is Halliburton; you’ll find the details in Global IPv6 Strategies: From Business Analysis to Operational Planning book (my review).

The need for IPv6 deployment is one of the topics discussed in the Enterprise IPv6 Deployment workshop. You can attend an online version of the workshop or we can organize a dedicated event for your team.

Apart from huge corporations using IPv6 internally, there are three groups pushing the public IPv6 deployment:

Service providers that are running out of IPv4 address space (which is limiting their growth opportunities);
Developers of new peer-to-peer applications that would like to get rid of NAT;
Consumer applications (like smart house) that would like to use globally-valid addresses. For example, Arch Rock is one of the companies making IPv6-based wireless sensor networks (a Google search for “IPv6 sensors” will also give you numerous sources).

For everyone else, IPv6 is something that you don’t want to be involved in (apart from the new toys perspective), but you’ll be forced to deploy it sooner or later.

Cisco IOS DHCP client behavior

2009-11-19T06:55:00.000+01:00

To complete the information about the DHCP address change behavior, I’ve collected a few more debugging printouts and combined them with the information from the previous posts into the Cisco IOS DHCP client behavior section of the DHCP client address change article in the CT3 wiki.

Detect short bursts with EEM

2009-11-18T07:01:00.001+01:00

Last week I’ve described how you can use EEM to detect long-term interface congestion which could indicate denial-of-service attack. The mechanism I’ve used (the averaged interface load) is pretty slow; using the lowest possible value for the load-interval (30 seconds) it takes almost a minute to detect a DOS attack (see below).

If you want to detect outbound bursts, you can do better: you can monitor the increase in the number of output drops over a short period of time.

Obviously you cannot use this mechanism to detect inbound (potentially DOS-related) floods as the drops occur on the Service Provider’s edge router.

Various polling and averaging options, including in-depth discussion of hysteresis, are covered in the Embedded Event Manager (EEM) workshop. You can attend an online version of the workshop or we can organize a dedicated event for your networking team.

More details: Interface load increases slowly after a flood starts

The interface input/output rates in bps or pps are computed as weighted averages using a sampling interval configured with load-interval command. To measure the delay between the start of a UDP burst and the denial-of-service alert generated by EEM (which is triggered by output bps rate exceeding ~80% of configured interface bandwidth), I’ve configured an access-list that reported the start of the burst and attached it to the outgoing interface:

ip access-list extended UDP
 permit udp any any gt 0 log
 permit ip any any
!
interface Serial0/1/0
 bandwidth 1000
 ip address 10.0.1.1 255.255.255.252
 ip access-group UDP out
 encapsulation ppp
 load-interval 30

It took the router almost a minute to report the interface saturation when the EEM applet was triggered by the TXload interface variable:

rtr#
00:27:38: %SEC-6-IPACCESSLOGP: list UDP permitted udp 
  10.0.0.10(1070) -> 10.0.20.3(5002), 1 packet
00:28:25: %HA_EM-6-LOG: IntOverload: Interface Serial0/1/0 
  overloaded: txload = 209

The OutputDrops EEM applet which relies on the output_packets_dropped variable detected the flood within 10 seconds.

Challenge: OSPF neighbor changing state from DOWN to DOWN

2009-11-17T06:52:00.002+01:00

I’ve had an interesting discussion with Nicolas who optimized my OSPF neighbor loss EEM applet assuming the OSPF-5-ADJCHG message reports only OSPF neighbor state transitions from DOWN to FULL and from FULL to DOWN. I knew I'd seen stranger messages in my lab and was able to produce these ones after fumbling with OSPF configurations of two routers connected with a serial link:

*19:34:42.765: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.1.1 on Serial1/0 from →
  EXSTART to DOWN, Neighbor Down: Too many retransmissions
*19:35:42.773: %OSPF-5-ADJCHG: Process 1, Nbr 10.0.1.1 on Serial1/0 from →
  DOWN to DOWN, Neighbor Down: Ignore timer expired

The messages are repeated approximately every three minutes (using the default OSPF timers).

Here's the challenge: what was going on and how was I able to produce these messages?

HQF: intra-class fair queuing

2009-11-16T06:43:00.001+01:00

Continuing from my first excursion into the brave new world of HQF, I wanted to check how well the intra-class fair queuing works. I’ve started with the same testbed and router configurations as before and configured the following policy-map on the WAN interface:

policy-map WAN
 class P5001
    bandwidth percent 20
    fair-queue
 class P5003
    bandwidth percent 30
class class-default
    fair-queue

The test used this background load:

Class	Background load
P5001	10 parallel TCP sessions
P5003	1500 kbps UDP flood
class-default	1500 kbps UDP flood

As expected, the bandwidth distribution between the three traffic classes was almost optimal:

a1#show policy-map interface serial 0/1/0 | include map|bps
    Class-map: P5001 (match-all)
      30 second offered rate 394000 bps, drop rate 0 bps
      bandwidth 20% (400 kbps)
    Class-map: P5003 (match-all)
      30 second offered rate 2073000 bps, drop rate 1479000 bps
      bandwidth 30% (600 kbps)
    Class-map: class-default (match-any)
      30 second offered rate 1780000 bps, drop rate 790000 bps

Next I’ve started a single (iperf) TCP session in the P5003 class. Using traditional CB-WFQ the session wouldn’t even start due to heavy congestion caused by the UDP floods. The same problem occurred with HQF since the P5003 class was using FIFO queuing.

However, once I’ve configured fair-queue within the P5003 class, the TCP session got half the allocated bandwidth:

$ iperf -c 10.0.20.10 -t 3600 -p 5003 -i 60
------------------------------------------------------------
Client connecting to 10.0.20.10, TCP port 5003
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[1916] local 10.0.0.10 port 1309 connected with 10.0.20.10 port 5003
[ ID] Interval       Transfer     Bandwidth
[1916]  0.0-60.0 sec  2.05 MBytes   287 Kbits/sec
[1916] 60.0-120.0 sec  2.05 MBytes   286 Kbits/sec

As expected, if you start numerous parallel TCP sessions, each one will get as much bandwidth as the UDP flooding stream. I started ten parallel TCP sessions with ipref and got an aggregate goodput of 524 kbps (leaving UDP flood with approximately 60 kbps):

$ iperf -c 10.0.20.10 -t 3600 -p 5003 -i 60 -P 10
------------------------------------------------------------
Client connecting to 10.0.20.10, TCP port 5003
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
[1916] local 10.0.0.10 port 1310 connected with 10.0.20.10 port 5003
[1900] local 10.0.0.10 port 1311 connected with 10.0.20.10 port 5003
[1884] local 10.0.0.10 port 1312 connected with 10.0.20.10 port 5003
[1868] local 10.0.0.10 port 1313 connected with 10.0.20.10 port 5003
[1852] local 10.0.0.10 port 1314 connected with 10.0.20.10 port 5003
[1836] local 10.0.0.10 port 1315 connected with 10.0.20.10 port 5003
[1820] local 10.0.0.10 port 1316 connected with 10.0.20.10 port 5003
[1804] local 10.0.0.10 port 1317 connected with 10.0.20.10 port 5003
[1788] local 10.0.0.10 port 1318 connected with 10.0.20.10 port 5003
[1772] local 10.0.0.10 port 1319 connected with 10.0.20.10 port 5003
[ ID] Interval       Transfer     Bandwidth
[1868]  0.0-60.0 sec   384 KBytes  52.4 Kbits/sec
[1820]  0.0-60.0 sec   384 KBytes  52.4 Kbits/sec
[1772]  0.0-60.0 sec   384 KBytes  52.4 Kbits/sec
[1836]  0.0-60.0 sec   384 KBytes  52.4 Kbits/sec
[1916]  0.0-60.0 sec   384 KBytes  52.4 Kbits/sec
[1900]  0.0-60.0 sec   384 KBytes  52.4 Kbits/sec
[1852]  0.0-60.0 sec   384 KBytes  52.4 Kbits/sec
[1804]  0.0-60.0 sec   384 KBytes  52.4 Kbits/sec
[1788]  0.0-60.0 sec   384 KBytes  52.4 Kbits/sec
[1884]  0.0-60.0 sec   384 KBytes  52.4 Kbits/sec
[SUM]  0.0-60.0 sec  3.75 MBytes   524 Kbits/sec

This is one of the very valid reasons Service Providers hate peer-to-peer file sharing services like BitTorrent.

Detect DOS attacks with EEM

2009-11-13T07:29:00.002+01:00

Someone sent me an interesting question a while ago: “is it possible to detect DOS flooding with an EEM applet?” Of course it is (assuming the DOS attack results in very high load on the Internet-facing interface) and the best option is the EEM interface event detector.

Interface event detector is just one of the many topics covered in the Embedded Event Manager (EEM) workshop. You can attend an online version of the workshop or we can organize a dedicated event for your networking team.

The interface event detector is more user-friendly than the SNMP event detector. You can specify interface name and parameter name in the interface event detector; with SNMP event detector you have to specify SNMP object identifier (OID). The interface event detector stores the interface name, measured parameter name and its value in three convenient environment variables that you can use to generate syslog messages or alert the operators via e-mail.

Notes:

You must use the bandwidth command to set the interface bandwidth to the actual line speed.
Set the bandwidth to the access speed of your Internet service on Ethernet uplinks.
The range of the rxload and txload parameters is between 0 and 255.
Interface load is computed as 256 * input-or-output-rate / configured-bandwidth.
The input-or-output-rate is a weighted average computed over the load-interval.

IPv6 in the Data Center: is Cisco ready?

2009-11-12T06:49:00.024+01:00

With the recent Cisco’s push into the Data Center environment and all the (not so very unreasonable) fuss around IPv4 address depletion and imminent need for IPv6, I wanted to check whether an all-Cisco shop could do the first step: deploy IPv6 on Internet-facing production servers. If you follow the various design guidelines, your setup will have at least the following elements (and I bet someone from Cisco has already told you that you also need XML firewall, Ironport and WAAS appliance):

Now let’s see how well these boxes support IPv6.

I’m describing the Data Center IPv6 deployment issues in the Enterprise IPv6 Deployment workshop. The diagram above was taken straight from the workshop materials.

Routers and switches (both Catalysts running SXI release and Nexuses running latest NX-OS) support everything you need, including IPv6 layer 3 virtualization (IPv6 in VRF, also known as 6vPE).

I couldn’t find 6vPE support in IOS XE Release 2, but then you’d most likely need it on the core switches, not on the edge routers.

ASA is a bit of a problem – the current software release does not support failover configurations with IPv6. The situation rapidly worsens as you go deeper into the Application Networking Services. I wasn’t able to find any mention of IPv6 in ACE, XML Gateway or WAAS configuration guides. If I’ve missed something, please let me know.

The status of IPv6 support in various Data Center components is summarized in the following table:

Equipment	Level of IPv6 support
Routers	Yes (6vPE on IOS XE might be missing)
Firewalls (ASA)	No redundancy (IPv6 failover doesn’t work)
Data center switches	Yes (Catalyst and Nexus)
Firewall Service Module (FWSM)	Not in transparent mode, on the main CPU (awfully slow) in routed mode.
Load balancers (ACE)	No
Application-level firewall (XML Gateway)	No
WAN optimization (WAAS)	No
Ironport	No

First HQF impressions: excellent job

2009-11-11T07:19:00.001+01:00

Several readers told me that the Hierarchical Queuing Framework introduced in IOS releases 12.4(20)T and 15.0 (why do I always have the urge to write 12.5?) works much better than CB-WFQ. After spending several hours trying to break HQF, I have to concur with them: Cisco’s engineers did a splendid job. However, the HQF behavior might be slightly counterintuitive to those that became too familiar with CB-WFQ.

For example, faced with this configuration …

policy-map WAN
 class P5001
    bandwidth percent 20
    fair-queue
 class P5003
    bandwidth percent 30
 class class-default
    fair-queue

… one might assume that all three classes will get a proportional part of the remaining bandwidth (50%). Not true. An HQF class with a bandwidth allocation gets as much as it’s asked for. It might get more, but if all classes are fully congested, the remaining bandwidth is distributed equally between classes without explicit bandwidth allocation.

When I ran 30 parallel TCP sessions across a 2 Mbps link (10 TCP sessions in each class), I’ve got these results:

a1#show policy-map interface serial 0/1/0 | include map|bps
    Class-map: P5001 (match-all)
      30 second offered rate 418000 bps, drop rate 21000 bps
      bandwidth 20% (400 kbps)
    Class-map: P5003 (match-all)
      30 second offered rate 613000 bps, drop rate 15000 bps
      bandwidth 30% (600 kbps)
    Class-map: class-default (match-any)
      30 second offered rate 997000 bps, drop rate 0 bps

As you can see, all the remaining bandwidth was used by the best-effort class-default.

Testbed

I was performing the QoS tests on a 2Mbps PPP link between two 2800-series routers running IOS release 15.0(1)M. The relevant parts of the router configuration are shown below.

Access lists permit TCP and UDP traffic to the same port. I needed a mix of TCP and UDP to test intra-class queuing behavior.

ip access-list extended P5001
 permit tcp any any eq 5001
 permit tcp any eq 5001 any
 permit udp any any eq 5001
 permit udp any eq 5001 any
ip access-list extended P5002
 permit tcp any any eq 5002
 permit tcp any eq 5002 any
 permit udp any any eq 5002
 permit udp any eq 5002 any
ip access-list extended P5003
 permit tcp any any eq 5003
 permit tcp any eq 5003 any
 permit udp any any eq 5003
 permit udp any eq 5003 any

Class maps:

class-map match-all P5001
 match access-group name P5001
class-map match-all P5003
 match access-group name P5003
class-map match-all P5002
 match access-group name P5002

Interface configuration:

interface Serial0/1/0
 bandwidth 2000
 ip address 10.0.1.1 255.255.255.252
 encapsulation ppp
 ip ospf 1 area 0
 load-interval 30
 !
 service-policy output WAN

I used iperf to generate TCP load and my own flood.pl to generate UDP load. The following command was used to start iperf:

$ iperf -c host -t 3600 -p port -i 60 -P 10

Solution: Bandwidth+Police actions in CB-WFQ

2009-11-10T07:28:00.009+01:00

Most of the respondents to my last week’s challenge got it almost right. The minor (common) error was the assumption that police rate percent 50 would result in a TCP session getting 50% of the bandwidth. Eyal got that right: the TCP throughput is always significantly lower than that due to frequent drops caused by low burst sizes assumed by the police command and resulting TCP restarts (the most I was able to push through was around 90 kbps; half of the bandwidth would be 128 kbps).

Many respondents got the third case (bandwidth class, police class and default-class all active at the same time) wrong. Vaidotas was guessing in the right direction and Petr knows the correct answer, but did not want to spoil the fun. Here’s the surprising result: the bandwidth class gets almost all the bandwidth. Sometimes the TCP sessions in other classes wouldn’t even start.

To understand that behavior, we’d need to go deep into the bowels of Weighted Fair Queuing … but I have to deliver my presentation first. In the meantime, enjoy a wonderful in-depth article written by Petr Lapukhov.

Recommendation

If a single class in an outbound service-policy uses the bandwidth action, all the other classes should use the bandwidth action as well. The classes without the bandwidth action and the default class might get starved during congestions.

ITU: Grabbing a piece of the IPv6 pie?

2009-11-09T07:06:00.000+01:00

ITU (the organization formerly known as CCITT) is having a bit of a relevance problem these days: its flagship technological achievements, including X.25, ISDN, ATM and SDH are dead or headed toward oblivion … and a former pariah, a group of geeks, is stealing the show and rolling out the Internet. No wonder their bureaucrats are having a hard time figuring out how to justify their existence. For years they’ve been lamenting how much they’ve contributed to the Internet (highly recommended reading for Monty Python fans) and how their precious contributions were unacknowledged. Now they came forth with a “wonderful” idea: the history of IPv4 address allocation proves that the wealthy nations and early adopters managed to grab disproportionate parts of the IPv4 address space (well, that’s true), so they made it their mission to protect the poor and underdeveloped countries in the brave new IPv6 world. In short, they want to become an independent address allocation entity (RIR). It looks like another worldwide bureaucracy is exactly what we need on top of all the other problems we have with IPv6 deployment.

Obviously there’s no need for ITU-operated RIR, it’s a too-obvious power grab plot, but it finally pushed NRO, ICANN (and all regional registries) to develop a fact sheet explaining the relevant IPv6 facts and address allocation process in layman terms. This document was forwarded to ITU together with an “interesting” cover letter (a must-read).

If you want to know the gory details, read the transcript notes of the RIPE-59 meeting (the interesting parts start in the second half with the presentation by Nick Thorne)

However, this is not a new development. Their quest started years ago with two ideas: either independent national registries or ITU-as-a-registry (see the Competition in IPv6 Addressing report) and was reignited this year when it became obvious widespread IPv6 deployment is imminent. The “IPv6 Public Policy Considerations” presentation by ITU Deputy Secretary General is full of subtle and not-so-subtle hints about the need for deeper ITU involvement in the IPv6 deployment process.

You simply have to go through that presentation. Even if you don’t want to know its contents, its copious selection of colors and fonts will enhance your inner artistic self (Health warning: prolonged exposure might cause uncontrolled laughter). One also has to wonder whether ITU sees WWW as anything more than a delivery mechanism for Microsoft Office files.

The whole thing would be laughable if it wouldn’t be dead serious: information like the above-mentioned presentation is distributed to national members of ITU. If these members have limited exposure to the actual IPv6 issues, they might act according to the ITU recommendations … and this is where you can step in: spread the word. Educate. Talk about IPv6. Explain it … but keep it simple ;)

Last but not least, hat tip to Jan Žorž for bringing this issue to my attention. If only he’d finally realize he needs to write a bilingual blog …

Overloaded …

2009-11-09T07:05:00.001+01:00

Last week I’ve published two posts that deserve a follow-up/summary. Don’t worry, it’s coming. I’ve been extremely busy working on a customized version of the “Market trends in Service Provider networks” presentation that I’m delivering tomorrow … and I’ve managed to stumble across two “interesting” topics involving ITU; the first one is described in the next post and the second one needs a bit more investigation.

Challenge: CB-WFQ Bandwidth+Police behavior

2009-11-06T07:24:00.004+01:00

I have to admit I was somewhat surprised by the lab test results I’ve published in my previous CB-WFQ post. It looks like we’ve been fed misleading information about (classic) CB-WFQ behavior for years.

Don’t tell me that things are completely different with HQF implemented in IOS releases 12.4(late)T and 15.0. I know that … but 95+% of the installed base do not use those releases.

Let’s see whether you can figure out what my next lab test results showed. I’ve been running three parallel TTCP sessions on ports 5001, 5002 and 5003 across a 256 kbit point-to-point link. Here’s the relevant part of my router configuration:

policy-map WAN
 class P5001
  bandwidth percent 50
 class P5002
   police rate percent 50
     violate-action drop
!
interface Serial0/1/0
 bandwidth 256
 clock rate 256000
 ip address 10.0.6.1 255.255.255.252
 encapsulation ppp
 ip ospf 1 area 0
 load-interval 30
 service-policy output WAN

TTCP is a program that sends meaningless data across a TCP session. It’s a nice load to use in a QoS test, as it uses TCP stack (ensuring it behaves like a real application) but still sends the data as fast as possible (as it spends no time generating it).

What would be your answers to these questions?

A single TTCP session is running on port 5001. How much bandwidth does it get?
A single TTCP session is running on port 5002. How much bandwidth does it get?
Three parallel TTCP sessions are running on ports 5001, 5002 and 5003. How much bandwidth does each session get?

Try to find the answer before reading the solution.

Off-topic: universal engineers

2009-11-05T06:18:00.000+01:00

Decades ago when I was still in high school and working on a programming project during the summer break, an IT old-timer gave me the following bit of advice: “Remember, God created professions so that everyone could do the job he’s qualified to do”. It took me years before I understood what he had been trying to tell me, but this seems to be an industry-wide disease. Judging by some of the e-mails I receive a lot of people who are proficient in other IT specialties think they can configure the routers with a little help of good uncle Google and free support from fellow bloggers.

It seems the “ability” of a “generic” IT employee to tackle any problem somewhat related to IT is also unique to our industry. Last week a woodworker was installing my kitchen and flatly refused to connect the electric cable of the ceramic cooktop to the wall outlet citing potential liabilities (please remember: I’m not living in US but in Central Europe). An HTML programmer asked to configure the enterprise firewall might not be so reluctant. Why do you think some people in our industry believe they are universal engineers?

CB-WFQ misconceptions

2009-11-04T06:58:00.003+01:00

Reading various documents describing Class-Based Weighted-Fair-Queueing (CB-WFQ) one gets the impression that the following configuration …

class-map match-all High
 match access-group name High
!
policy-map WAN
 class High
  bandwidth percent 50
!
interface Serial0/1/0
 bandwidth 256
 service-policy output WAN
!
ip access-list extended High
 permit ip any host 10.0.3.1
 permit ip host 10.0.3.1 any

… allocates 128 kbps to the traffic to/from IP host 10.0.3.1 and distributes the remaining 128 kbps fairly between conversations in the default class.

I am overly familiar with weighted fair queuing (I was developing QoS training for Cisco when WFQ just left the drawing board) and was thus always wondering how they manage to implement that behavior with WFQ structures. A comment made by Petr Lapukhov re-triggered my curiosity and prompted me to do some actual lab tests.

The answer is simple: CB-WFQ does not work as advertised.

To prove this claim, I’ve started two parallel TTCP sessions: one to IP address 10.0.3.1, the other to IP address 10.0.3.2. This is what show policy-map interface command displayed after a minute:

a1#show policy-map int ser 0/1/0
 Serial0/1/0

  Service-policy output: WAN

    Class-map: High (match-all)
      5996 packets, 3424386 bytes
      30 second offered rate 200000 bps, drop rate 0 bps
      Match: access-group name High
      Queueing
        Output Queue: Conversation 73
        Bandwidth 50 (%)
        Bandwidth 128 (kbps)Max Threshold 64 (packets)
        (pkts matched/bytes matched) 5981/3421234
        (depth/total drops/no-buffer drops) 4/0/0

    Class-map: class-default (match-any)
      516 packets, 270445 bytes
      30 second offered rate 6000 bps, drop rate 0 bps
      Match: any

The printout clearly demonstrates that the TCP session in the High class got way more than its allocated share while the TCP session in the class-default got 30 times less bandwidth.

Conclusion

The conversations in the class-default are treated as low-priority conversations and get significantly less bandwidth than other traffic classes.

Another successful IPv6 presentation

2009-11-03T07:04:00.002+01:00

Last week I’ve delivered another very successful IPv6 Deployment in Enterprise Networks presentation. It’s amazing how far into the interesting details we usually get even though the presentation is purposefully a high-level one. This time we’ve been discussing the liabilities the Service Providers might get exposed to when using Large Scale NAT and whether the enterprise networks using HTTP proxies for Internet access could avoid the migration to IPv6.

If you’re working in a large enterprise network and think your IT team could benefit from this presentation or associated IPv6 workshop, get in touch with me.

Book review: Foundation of Green IT

2009-11-02T06:47:00.002+01:00

If you want to understand the real impact of the recent Data Center hype without getting pulled into the technology morass the vendors so copiously spread in their white papers, read the Foundation of Green IT book published by Prentice Hall. Its author, Marty Poniatowski, uses two case studies to illustrate enormous savings that can be realized through server and storage consolidation. I loved the first half of the book: the author avoids the technology issues (I loved the introduction to RAID: “I do not cover RAID background … the Internet has a wealth of information on RAID”) and uses real-life data gathered in actual project to illustrate the savings. Each case study has several chapters, ranging from starting point discovery through implementation plans and ROI analysis; exactly what you need if you’re considering going down the data center redesign path. The “Desktop Virtualization” and “Data Replication and Disk Technology Advancements” chapters are thrown in for good measure.

The author makes the server and storage consolidation case studies even more interesting by describing actual products/solutions and inserting screenshots of actual reports throughout the text. Not surprisingly, he’s describing what he knows best: HP servers, EMC storage and VMware virtualization; a clear indication how far Cisco has to go to win the hearts and minds of the data center market.

After the fantastic first half, the second half is pure disappointment. It starts with three networking chapters that clearly show networking is not the author’s strong point. All of a sudden, the interesting real-life details are replaced with technology introductions … although the author quotes an IDC report in the preface claiming that the technology represents #9 (out of 10) challenge in data center environments. You’re pulled into quagmire of IOS release numbering and PSIRT advisories; the contrast with the excellent first half is all too obvious. There’s also no ROI analysis. No surprise: the author advocates replacing 10 obsolete low-end Fast Ethernet switches with a fully redundant Catalyst ~~6507~~C4507R-E; I’d love to see a ROI analysis that can justify that investment based on power and cooling savings.

The last part is a hodgepodge of topics. I was looking forward to the SQL Server Consolidation chapter; more so as it was written by a Microsoft’s engineer. It was boring and could be summarized in a single sentence: SQL Server is another application that can be virtualized using the rules outlined in the first five chapters.

The Green Data Center chapter is undoubtedly the best part of the second half of the book; it succinctly describes the issues of cooling, air flow, data and power cabling … but then the quality starts sliding down with Cloud Computing (no wonder; I have yet to find someone who can write something I will find interesting about this vaporware architecture) and Simple Power Savings (summary: turn off your computer at night) chapters. The last chapter; Managed Services: Remote Monitoring is a pure product description of a software product the author’s company is selling.

Regardless of the shortcomings of the last chapters (which were probably thrown in to expand the scope; a better title would be Foundation of Green Data Center), I can strongly recommend the book for its excellent descriptions of server virtualization and storage consolidation issues. I also have to congratulate its author for his marketing prowess; the whole book is a great case study/white paper for the services his company offers … and we’re willing to pay for the privilege to read it.

Hierarchical Queueing Framework: queue limits and output drops

2009-10-30T07:25:00.000+01:00

The QoS behavior in Cisco IOS has changed significantly with the introduction of the Hierarchical QoS Framework (HQF) in IOS release 12.4(20)T. Cisco is slowly producing in-depth articles describing the changes; the first one I’ve found documents the old and new output queue limits and output drops.

I’ve also added the link to this article to the Further reading section of my Queuing principles in Cisco IOS article.

IPv6 labs are no longer available on Cisco’s Partner Education Connection

2009-10-29T20:46:00.002+01:00

Csaba asked me about the availability of our IPv6 remote labs on Partner Education Connection. He wrote:

I was happy when I found this information, but after a while I realized that IPv6 labs are no more available on PEC site. Do you know any details why this topic has been removed from PEC or it can be that IPv6 is still there just I couldn't find it?

Cisco has removed almost all external remote labs from PEC in October. To increase the confusion, some of them might still appear in the catalog, but you cannot start them. The only thing we could do was to inform the users landing on our web site that the labs are no longer available and that they could buy them from our product catalog.

IPv6-capable or IPv6-ready: is it enough?

2009-10-28T06:49:00.000+01:00

During the IPv6 summit in Slovenia I’ve participated in a roundtable organized by our Ministry of Higher Education, Science and Technology. One of my points was that the government should require true IPv6 support in all its IT procurement processes to promote IPv6 adoption (I have to admit I’ve borrowed a few ideas from Geoff Huston’s “Is the Transition to IPv6 a Market Failure?” article) … and one of the participants (coming from the Service Provider industry) answered that “that’s common hygiene”. I’m not so sure.

Topics like this are covered in my Enterprise IPv6 Deployment workshop. Learn more about my workshops from my web site.

To start with, vendors will tell you that their products are “IPv6-capable” or “IPv6-ready”. The first term is (in my opinion) pure vaporware. Anything that was produced in the last 10 years can be made to run IPv6 (unless it’s a consumer product using severely limited hardware), so the “IPv6-capable” label is meaningless. For example, Cisco’s ACE Web Application Firewall seems to be IPv6-capable (translated from marketese: I haven’t been able to find a single occurrence of IPv6 in its administration or user’s guide, but I’m positive a software running on a Linux platform can be adapted to support IPv6).

IPv6-ready is more meaningful; it implies that the product actually runs IPv6. However, you should ask (at least) these two questions:

Does IPv6 support the same functionality as IPv4? For example, split DNS support in Cisco IOS works only with IPv4 DNS servers.
Does the product perform IPv6 operations with the same (or similar) speed as IPv4 operations?

The second question is particularly relevant for hardware-based routers and switches (one cannot imagine an application having significantly different performance when running over IPv6). These devices usually rely on ASICs and TCAM to perform high-speed packet forwarding. The TCAM width (number of bits available in a single TCAM entry) can significantly limit the wire-speed IPv6 capabilities. For example, PFC on a Catalyst 6500 cannot filter on full 128-bit IPv6 addresses and TCP/UDP port numbers.

Followup: What’s wrong with the Zone-Based Firewalls book

2009-10-27T07:00:00.000+01:00

I’d like to thank all the readers that took time and responded to my question about the failure of my Deploying Zone-Based Firewalls book. The sad short conclusion is: while everyone would love to have an electronic copy of the book, the technology and the mindsets are simply not ready yet. Here are the details:

You hate DRM. So do I. Every DRM scheme treats you like a common criminal that has to be severely restricted so that he can do the least damage to the “high-value” property the DRM is trying to protect (not to mention the risks of DRM failures, sellers going bankrupt etc.). For example, the Adobe Digital Editions (if it works at all … I was never able to get it running properly on my laptop) allow you to install the same book on a few machines, but you can’t carry it with you as you could a paper copy.

DRM on books is either too restricted or useless. As someone wrote in his comment, he prints the every digital book the moment he receives it (and you can always print it to a PDF printer). If the DRM protection doesn’t allow you to print the book, it’s probably too restricted; if it does, it’s useless.

Everyone loves downloadable PDF. Pirates included. Let’s be realistic: if you can view protected PDF files within an open-source viewer on Linux (and copy “protected” material from them), anyone with enough time can unprotect them, remove downloading user indication (PDF is not so hard to modify) and post them.

Safari is no protection. The digital books available on Safari in HTML and PDF format can be pirated as easily as anything else. For example, I’ve found the complete MPLS and VPN Security book from Cisco Press available “for free” in HTML format. I doubt Cisco Press released it that way; it was probably scraped from Safari (or some other HTML source).

Other issues

The fact that Cisco was giving away copies in non-DRM PDF format, the stealth marketing the book received (numerous comments indicated the readers using the zone-based firewalls were not aware the book exists) and problems with the ordering platform (you can’t order the DRM-protected PDF from Amazon, you have to do it through Cisco Press web site) didn’t help either.

The title of this post is “what’s wrong with my book”, so I’ll stop here. Many readers made very positive comment and offered useful suggestions that I’ll definitely follow. I would also recommend that you buy your copy of the book on Kindle or through Safari to avoid the Adobe DRM hassle.

What’s your take on Alcatel-Lucent IP+Optical integration?

2009-10-26T06:47:00.001+01:00

Approximately a month ago Alcatel-Lucent launched Converged Backbone Transformation (are they sharing marketing wizards with Cisco … or is the excessive hype an industry-wide phenomenon?): a visionary(?) convergence of IP and optical technologies. If you haven’t heard about it yet, you could try to start with the IDC report published on Alcatel-Lucent’s web site (I’m always amazed how some people manage to tell so little in so many words).

Once you get past the fluff to the details, it could be that they're implementing a lot of common-sense. For example, it looks like the lambda-level grooming replaces GBIC/SFP transceivers with something that can generate multiple lambdas on the router and feed these lambdas directly into the DWDM gear. In my understanding, it replaces the GE port-GBIC-fiber-GBIC-GE port-lambda generation-DWDM chain with the shorter and cheaper GE port-lambda GBIC-fiber-lambda port-DWDM chain (obviously, I might be completely wrong; it’s hard to deduce the details from a press release).

Anyhow, I’d really appreciate your thoughts on this launch. Does it make sense? How does it compare to what Cisco and Juniper are doing? Is this a move in the right direction … or is Alcatel-Lucent playing a catch-up and trying to cover it with a grand marketecture?

IOS packaging: Moore’s Law Won

2009-10-23T06:30:00.003+02:00

Great news: Cisco launched a new series of midrange routers on Tuesday. They're very probably great products (I wouldn't expect less from Cisco). Also as expected, their marketing department couldn’t help itself (yet again) and had to position the launch as a universe-changing event: this time they Revealed the Borderless Network and spent loads of money producing “viral videos”. OK, maybe their average customer is stupid enough to fall for those tricks; I’m positive you’re not … so let’s see what’s really new (here's what Cisco admits is new after you've got past all the marketing fluff):

All the embedded “WAN” ports are Gigabit Ethernet uplinks. Good.
They claim up to 5-times higher performance than the previous routers. Average. The ISR series was launched in 2004 and Moore’s law predicts 5.8-times increase.
Lots of the old interface modules are supported. Amazing; I’m just hoping it doesn’t hurt the performance.
They’ve replaced the old half-hearted attempts to include an x86 generic application platform within a router with the Service Ready Engine (another great marketing invention … sounds so much better than a Linux blade) modules, having up to 4GB of RAM and 1TB of hard disk. I don’t want to know how the people who bought the old AXP platform feel reading these specs.

However, one of the most important changes from the network manager perspective is the unified IOS image. No more hassle in the image selection process: you have a single image that contains all the function you’ll ever need. If your quality control process requires IOS image lab tests, you can do it once regardless of which features you need. When you need to deploy additional functionality, you just order new license, enter it in the router and start using the new features.

I’m positive someone will try to position the unified IOS image as the next best thing since the invention of sliced bread. However, it’s yet again a simple result of Moore’s law; the IOS developers were not able to develop the code fast enough to keep with the dropping DRAM/Flash prices. All the memory needed for the “kitchen-sink” image of today costs less than the IP base image in days when IOS 12.2 was released.

Actually, the official name for the kitchen sink image is Advanced Enterprise Services, but those of you who have been in the business long enough know probably still remember what the –k- acronym in the IOS image name stood for.

To make my findings a bit more “scientific”, I’ve tried really hard to find out when individual IOS releases were launched and produced a graph comparing Flash image sizes of compressed high-end IOS images (ENTERPRISE/FW/IDS IPSEC 3DES or Advanced Enterprise Services) with the exponential curve predicted by the Moore’s law based on compressed 12.2 IP Plus image (the smallest image for the 7200-series router I was able to find).

The image sizes were taken from CCO download area. I’ve used the images for the 7204 router to eliminate the potential impact of changed CPU instruction set. In some cases, I was not able to find the “original” images, so I’ve assumed the mainline IOS image size did not increase significantly during its maintenance period.

Report interface loss based on OSPF neighbor loss

2009-10-21T06:00:00.004+02:00

Nicolas sent me an interesting problem: he has numerous point-to-point GRE-over-IPSec tunnels on his core router and detects remote site failure with OSPF neighbor loss events. He would like to receive an e-mail when an OSPF neighbor goes down (quite easy to do with EEM), but would also like to receive interface description in the e-mail subject to simplify his troubleshooting.

With the regular expressions available in EEM 3.0 you can extract interface name from syslog message, execute show interface command and extract the interface description from it.

The EEM applet source code is available in the CT3 wiki

This article is part of You've asked for it series.

Help appreciated: what’s wrong with my Zone-based Firewalls book?

2009-10-20T06:17:00.000+02:00

A quick question for you: in two years since my Deploying Zone-based Firewalls digital short cut (marketese for downloadable PDF) was published, we’ve sold around 200 copies of it. Obviously we’re doing something wrong and I’d appreciate your opinion: is it the topic (are you using ZB firewall on Cisco IOS?), the format (would you prefer paper copy?), the platform (Cisco IOS as a firewall), pricing ($14.99 for 112 pages) or something else?

What went wrong: TCP lives in the dial-up world

2009-10-19T06:36:00.000+02:00

As expected, my “the socket API is broken” post generated numerous comments, many of them missing the point (for example, someone scolded me for quoting Wikipedia and not the official Linux documentation). I did not want to discuss the intricate technical details of the various incarnations of the API but the generic stupidity of having to deal with low-level networking details in the application.

Fabio was kind enough to provide the recommended method of using the Socket API from man getaddrinfo, effectively proving my point: why should every application use a convoluted function when all we want to do (in most cases) is connect to the server.

Patryk went even further and claimed that the socket API provides “basic functionality” and that libc is not the right place for anything more. Well, that mentality caused most of the IPv4-to-IPv6 application-related issues: obviously the applications developed before IPv6 was a serious consideration had to be rewritten because all the low-level code was embedded in the applications, not isolated in the library. A similar problem has effectively stalled SCTP deployment.

However, these are not the only problems we’re facing today. Even if the application properly implements the “try connecting to multiple addresses returned by DNS” function, the response time becomes unacceptable due to the default TCP timeout values coded in various operating systems’ TCP stacks.

For example, it takes up to three minutes for a TCP connect call to timeout on a Fedora-11 Linux distribution (the connect call aborts immediately if an intermediate router sends back an ICMP unreachable reply and the ARP timeout causes an abort in three seconds). Windows XP is slightly better; the default timeout is set at 20 seconds.

You might wonder what prompted the TCP designers to choose these exceedingly large values. TCP was designed more than 20 years ago when the analog dialup modems were commonly used to connect to the Internet. These modems could take a minute (or longer) to establish the connection and if you wanted to have a reliable TCP session setup, you had to wait significantly longer before aborting the session setup system call. The Internet has changed dramatically in the meantime, but nobody ever bothered changing the defaults.

If you want to rush and write a comment how the default can be changed, you’re yet again missing the point: we cannot implement multihomed IP hosts using more than one IP address due to the crazy default TCP timeout values. As soon as the first address becomes unreachable, the session establishment time (for an average user using out-of-box software) becomes unacceptable.