John H Goodwin

[Attendee1] So, how can I get one for $10/mo?

[@JonathanD] Attendee1: become an ibm business partner doing more than $100,000 a year in IBM sales, then apply for a development lease.

[Attendee2] Well, blade centers look interesting, but some of the information is moderately, er, confusing?

[@JonathanD] Anything in particular, Attendee2?

[Attendee2] Like, specifically how networking works with the backplane

[@JonathanD] I'll summarize briefly, then you can ask questions, ok?

[@JonathanD] in a normal configuration, we'll put 2 switch modules in a chassis.

[@JonathanD] These switches go in bays 1 and 2

[@JonathanD] regardless of the chassis

[@JonathanD] bays 1 and 2 are essentially hard wired to a network connection on each blade

[@JonathanD] each blade has to built in NICs

[@JonathanD] it's actually a single dual port NIC, physically

[@JonathanD] first port goes to bay 1

[@JonathanD] second port goes to bay 2

[@JonathanD] so thats the baseline. Each blade connects to bay 1 and 2 at least once.

[Attendee2] So, the switch in the bay determines the speed of networking between blades, not the speed of the backplane?

[@JonathanD] Attendee2: yes. The backplane is basically just wiretraces.

[@JonathanD] theoretical limit is 10GB

[@JonathanD] however, all blades, to date, have only 1GB nics built in

[@JonathanD] now, I said at least once, because a 2 slot blade has at least the potential for 2 connections to each switch

[@JonathanD] depending on the blade, how it's configured, and a few other things, you'll have 1 or 2, to each

[Attendee2] So, if you wanted to add 10gb or infiniband or whatever you'd have to put in a switch for it and a module in every blade

[@JonathanD] Attendee2: essentially yes

[@JonathanD] I'm going to speak briefly on infiniband and what it can offer in a moment

[@JonathanD] just high level.

[@JonathanD] now, the enterprise chassis (just called bladecenter) has 4 IO module slots

[@JonathanD] as we discussed , 1 and 2 are dedicated to ethernet

[@JonathanD] externally, they can be fiber or copper.

[@JonathanD] the other 2 can be a wide variety of things

[@JonathanD] fiber channel, SAS, more ethernet, infiniband

[@JonathanD] in the newer H chassis, there are all-told 10 slots for different IO options.

[@JonathanD] one of these is a high speed infiniband switch.

[@JonathanD] with the infiniband switch, you can do some interesting things...

[@JonathanD] not much support exists yet for infiniband, so what we have is a special adapter that sits in the blade and talks to the infiniband switch (preferably, 2 of them for redundancy, same as ethernet)

[@JonathanD] this gives us, on top of our 2 built in GBE connections, 6 additional "virtual" connections on infiniband fabric, to each blade.

[@JonathanD] with the infiniband switch, each of these 6 can be either Fiber channel OR 10GBE

[@JonathanD] and it can vary by blade.

[@JonathanD] Attendee2: does that answer your question, and do you have any more?

[Attendee1] Do cross-blade ethernet communications exit the ethernet I/O ports or does it handle ethernet requests between blades internally?

[Attendee1] Basically, can the ethernet be unplugged external to the bladecenter and still have ethernet between the blades themselves?

[@JonathanD] Attendee1: absolutely

[@JonathanD] the switches in a blade chassis are real switches, most of them with vlan capability

[Attendee3] Why do blades talk to each other? do they all pool into one server or something? like one blade can be a file server, one an email server, one a Web server, etc? sorry for a n00b question

[@JonathanD] but yes, often folks will put a substantial infrastructure inside a single chassis

[Attendee2] er, yeah it did (sorry thought I answered)

[@JonathanD] that doesn't need to exit the chassis at all except for the end user responses

[@JonathanD] so your web server, app server, and db server (or cluster) can live inside the chassis.

[Attendee3] So I could have like 20 blades, all doing different things, but it looks like one server to an end user? Wow.

[Attendee4] when i look at photos of the H Chassis it looks closed off from the back, so if i put an expansion module on a blade to put in a PCI card or something, where will the connection sit for that card ?

[@JonathanD] furthermore, several of the switches are capable of load balancing with no additional hardware.

[@JonathanD] Attendee4: there are fillers on the back of a BC H, you take them out to install a switch module

[@JonathanD] that switch module has external ports

[Attendee1] JonathanD: What's the network interconnection speed between blades?

[Attendee4] perfect !

[@JonathanD] the number and type of ports vary, based on what you choose.

[@JonathanD] Attendee1: up to 40Gbit per blade, depends entirely on the switches installed.

[Attendee4] i would just love, for redudnancy, to have my PBX sit on a blade, so i would need to add my T-1 interface card to the blade

[Attendee1] JonathanD: Does it only depend on the switch, or does the blade have to support that speed too?

[@JonathanD] Attendee1: the adapter in the blade does, yes.

[Attendee3] 40Gbit/sec... wows

[@JonathanD] much like a 1Gb switch does no good if you have a 10Mbit nic

[Attendee1] JonathanD: What's the oldest that still supports 40Gbit?

[Attendee1] (stock from IBM)

[ttender3 still has 100Mbit network...with some nodes only 10Mbit

[@JonathanD] the HS21s, almost all of them, have support for infiniband to bays 7-10 which supports that 40GBit

[Attendee1] Also, does that interconnect show as a separate network card?

[Attendee3] Infiniband = unlimited speed?

[Attendee3] i.e. "infini"band?

[@JonathanD] Attendee1: that would be an infiniband connect, which in this case you would probably configure as virtual 10GbE adapters.

[@JonathanD] you can actually exceed that 40Gbit in some cases

[@JonathanD] and as the midplane is basically just traces, thats not a hard value.

[@JonathanD] it could increase as new switching technology becomes available.

[Attendee1] The 40Gbit is per blade or overall?

[@JonathanD] Attendee1: thats the current max that can be delivered to a single blade

[@JonathanD] within the chassis, that would be point to point.

[Attendee1] Is that full duplex or half duplex speed?

[@JonathanD] which makes it excellent for vmotion, for example.

[Attendee1] (as in utilizing both send/receive to get 40Gbps, or are both @ 40Gbps) ?

[@JonathanD] thats half, so you can push 40 in both directions.

[Attendee1] Nice.

[@JonathanD] across 4 interfaces or virtual interfaces.

[@JonathanD] in short, particuarly with infiniband, you have *lots* of bandwidth.

[Attendee3] I used to think of Mbits like mph...10 is slow and sucky and 100 was really fast.

[Attendee3] Now we have 40Gbit!?

[Attendee3] Is this available for regular non-blade servers?

[Attendee3] the most I've seen is 1Gbit, honestly.

[@JonathanD] Attendee3: infiniband is, yes. I don't know if the virtual IO tech is.

[Attendee1] So the virtual network adapter you mentioned. How is this configured?

[Attendee3] "the virtual IO tech" means what?

[Attendee1] You install a driver in Windows, and configure this on the blade center level?

[@JonathanD] Attendee1: my understanding is that it's configured in the cards BIOS

[@JonathanD] so you can do it pre-install

[Attendee1] I see.

[@JonathanD] and much like the qlogics, once the machine is up you can install software to manage it without a reboot.

[@JonathanD] it can also be assigned from within the managemnet module, and even scripted.

[Attendee1] So you could have upto 4 of these virtual cards or can you over allocate the bandwidth, and it'll do best effort routing?

[Attendee1] (Per blade)

[@JonathanD] it will drive to the limit of the infiniband, yes.

[@JonathanD] you can configure 10GbE interfaces, and 4GB FC interface.

[Attendee1] So other than OS limits, how many network interfaces can you create?

[Attendee1] Per blade.

[@JonathanD] 8

[@JonathanD] includeing the 2 built in

[@JonathanD] 6, within infiniband

[Attendee3] and what pretel do you *do* with 6 virtual NICs?

[Attendee1] So, each blade has 2 which are connected to the external ethernet I/O ?

[Attendee3] Attendee1: sounds like it

[@JonathanD] Attendee1: to switch bays 1 and 2, yes.

[@JonathanD] the "default" ethernet.

[@JonathanD] those are part of the system board and non-removable.

[Attendee1] What's the max interconnect speed of HS20's ?

[@JonathanD] akin to the built in nics on an xseries.

[@JonathanD] Attendee1: depends :)

[Attendee3] do these suckers have PCIe or AGP support?

[@JonathanD] if you have fiber channel, you'll be using your 1 and only expansion slot for that.

[@JonathanD] Attendee3: as they don't have slots like that, no. But at a technical level, yes.

[Attendee1] Attendee3: It's not designed for high-speed graphics cards.

[Attendee1] (for games).

[@JonathanD] the adapters in them are PCI-X or PCI express, in the newer

[@JonathanD] to answer Attendee4's question

[Attendee3] JonathanD: ahh okay

[@JonathanD] you can purchase a sidecar, that takes a slot

[@JonathanD] which allows you to install 1 or 2 PCI cards.

[@JonathanD] these would be accesible at the front of the blade chassis.

[@JonathanD] Attendee4: it's an uncommon need, only there for rare uses.

[Attendee4] i dont know if i want this nice "all-in-one" enclosure to have 2 cables running off the front of the chassis

[@JonathanD] I have to wind this up in the next few minutes.

[@JonathanD] any more questions, right now?

[Attendee1] JonathanD: Okay, so HS20, I'm still not sure what the interconnect speed is between blades.

[@JonathanD] Attendee1: ahh, sorry.

[@JonathanD] if you forgo fiber, you can put a 10GbE in modules 3 and 4

[Attendee1] Does the HS20 still do the virtual network thing too?

[@JonathanD] for 20GbE, + the 2 base.

[@JonathanD] Attendee1: afraid not, it does not have the slots for it.

[Attendee1] So modules 3 & 4 would be used internally, or what?

[@JonathanD] Attendee1: internally and externally

[@JonathanD] with 1 or more 10GbE connections to external

[Attendee1] Okay, a module services how many blades? All of them?

[@JonathanD] yes.

[@JonathanD] or has the potential to

[@JonathanD] if, for example, you have san, and one doesn't need san

[Attendee1] So redundancy aside, you could just do one module for network, right?

[@JonathanD] you don't have to put an adapter in it.

[@JonathanD] Attendee1: yes, and some do.

[Attendee4] JonathanD: i think i need to know more about the storage, so for your next get together we should do that !!

[Attendee1] Yes, what's the recommended way to add storage to these things?

[Attendee3] I'm still 100% confused about virtual networking.

[@JonathanD] there was some talk of a hybrid card that would fit the existing slot in an hs20 and attach to the infiniband fabric.

[Attendee3] SCSI, I'd assume.

[@JonathanD] but there was not enough customer interest in it, unfortunatly.

[@JonathanD] Attendee3: doubtful.

[@JonathanD] Attendee1: I think we'll have to revisit storage.

[Attendee3] IEEE 1394?

[@JonathanD] I'm gonna have to get on a customer call.

[Attendee1] JonathanD: K.

[@JonathanD] Attendee3: FC or SAS

[Attendee4] well theres no way i could afford the fibre, so i will definately be going with the SAS

[Attendee1] JonathanD: Thanks.

[Attendee3] Bye.

[Attendee3] Yes, thank you JonathanD.

[@JonathanD] no problem.

[@JonathanD] hope this was helpful for all involved.