• Where did CKD disks come from?

    From John Levine@21:1/5 to All on Thu Dec 12 20:31:53 2024
    This is a question about I/O architecture rather than CPU architecture, but close enough.

    IBM's early disks all had fixed size sectors, or in one case I found, an option to
    write one big record per track if the computer had enough memory to buffer it.

    System/360 changed that to a much more complicated Count-Key-Data format where each track was formatted with records of arbitrary size (that's what the count was for) and records could have a hardware key. The keys were used for ISAM files, where the channel and device controller could search through a track or cylinder to find a record with a desired key.

    I can somewhat see the motivation, since that made it possible to make the records on the disk match the logical sizes that applications used, and the CPU could tell the channel and controller to do a key search while the CPU went off to do something else. But in practice, this turned out to be a bad idea. Nearly everyone had more than one disk attached to each controller which meant that if it was doing a key search on one drive, it couldn't do anything with the other drives. Originally, even if the disk was formatted with fixed sized records the channel and controller had to start at the track origin and search for the record number.

    Soon they added Rotational Position Sensing (RPS) which let the channel do other
    stuff until the drive got close to the desired position, and eventually they invented a new acronym FBA to describe the block format everyone else used and for many years CKD disks have been simulated in microcode on FBA devices. ISAM was replaced with VSAM KSDS which uses B-trees of fixed sized blocks.

    Even in the CKD era, at the low end, 360/20 and 1130 and 1800 had 2311 disk drives formatted with fixed sectors that simpler controllers could handle. The physical disk packs were the same but couldn't be used on CKD drives without being erased and reformatted.

    My question is where did CKD come from? I looked at the 1964 IBM Systems Journal
    issue that describes the design of S/360 which has an article describing the way
    the channels work, but nothing about the disks. Wikipedia has a long list of CKD
    drives and controllers but no hint of whose idea it was.

    One clear mistake is that they underestimated how quickly main memories would grow, which meant that you could cache keys in memory so most of what would otherwise be found by a channel ISAM search was in the cached keys and you could
    usually directly read correct record on the disk. Similarly, if you have
    plenty of memory to buffer multiple disk blocks, splitting logical records of whatever size across fixed size blocks is no big deal.

    --
    Regards,
    John Levine, johnl@taugh.com, Primary Perpetrator of "The Internet for Dummies",
    Please consider the environment before reading this e-mail. https://jl.ly

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From John Levine@21:1/5 to All on Fri Dec 13 02:01:10 2024
    According to Lynn Wheeler <lynn@garlic.com>:

    John Levine <johnl@taugh.com> writes:
    My question is where did CKD come from? I looked at the 1964 IBM Systems Journal
    issue that describes the design of S/360 which has an article describing the way
    the channels work, but nothing about the disks. Wikipedia has a long list of CKD
    drives and controllers but no hint of whose idea it was.

    One clear mistake is that they underestimated how quickly main memories would
    grow, which meant that you could cache keys in memory so most of what would >> otherwise be found by a channel ISAM search was in the cached keys and you could
    usually directly read correct record on the disk. Similarly, if you have
    plenty of memory to buffer multiple disk blocks, splitting logical records of
    whatever size across fixed size blocks is no big deal.


    big part of CKD was trade-off of I/O and memory. search & multi-track
    search ... have the I/O channel program search for the record you wanted
    ... at least by the mid-70s, the trade-off was inverting.
    [ long example of terrible CKD performance ]

    Any idea who invented CKD? The Pugh et al history book says a lot about
    disk hardware but nothing about software or CKD. Someone must have invented
    it but who? When?

    --
    Regards,
    John Levine, johnl@taugh.com, Primary Perpetrator of "The Internet for Dummies",
    Please consider the environment before reading this e-mail. https://jl.ly

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Lynn Wheeler@21:1/5 to John Levine on Thu Dec 12 15:39:15 2024
    John Levine <johnl@taugh.com> writes:
    My question is where did CKD come from? I looked at the 1964 IBM Systems Journal
    issue that describes the design of S/360 which has an article describing the way
    the channels work, but nothing about the disks. Wikipedia has a long list of CKD
    drives and controllers but no hint of whose idea it was.

    One clear mistake is that they underestimated how quickly main memories would grow, which meant that you could cache keys in memory so most of what would otherwise be found by a channel ISAM search was in the cached keys and you could
    usually directly read correct record on the disk. Similarly, if you have plenty of memory to buffer multiple disk blocks, splitting logical records of whatever size across fixed size blocks is no big deal.


    big part of CKD was trade-off of I/O and memory. search & multi-track
    search ... have the I/O channel program search for the record you wanted
    ... at least by the mid-70s, the trade-off was inverting.

    program libraries were typically partitioned datasets (PDS) with
    directory at front. Channel program would do multi-track search of PDS directory looking for a PDS directory program entry, read it ... then do
    a channel program to move arm to that head position and read the
    program. For each seach compare, the search CCW would refetch the match
    from processor memory ... for the duration of the search, the device, controller and channel would be locked.

    the architecture became heavily ingrained into the batch operating
    systems. around 1980, I offered to provide FBA support to them. I was
    told even if I provided fully integrated and tested, I still needed a
    couple hundred million in incremental sales to cover education and documentation for the changes ... and since they were already selling
    every disk made ... it would just change from CKD to FBA (with no
    incremental sales) ... also I couldn't use (FBA) life-time savings in
    the business case.

    late 70s, I had been brought into large datacenter for major national
    grocery store chain ... had multiple systems in loosely-coupled shared
    DASD configuration (stores group into multiple geographical regions
    mapped to different systems). They were having horrendous performance
    problems and most of the corporate specialists had been brought though
    before I was called.

    Had a classroom with tables covered with large paper piles of
    performance activity data from all the systems. After about 30mins I
    noticed that during worst performance periods, the aggregate I/O (summed
    across all the systems) peaked around 6-7/sec (3330, 19tracks/cyl, RPS)
    and asked what it was.

    It turns out it was shared disk (for all systems) that contained all
    store applications ... and it was basically caped at doing two program loads/sec for the hundreds of stores across the country. It had a 3cyl
    PDS directory and would avg of 1.5 cyl multi-track search for each
    application ... i.e. full cyl. multi-track search of 19tracks at
    60revs/sec (.317secs) followed by 9.5track (.16secs) ... during which
    time the disk was locked out for all systems and the controller (and all
    drives on that controller were locked out). Once the PDS entry was
    found&read, it could use the information to move the arm for
    reading/loading the program.

    Somewhat engrained in the professional minds was typical 3330 would do
    30-40 I/Os per sec ... individual system activity reports only showed
    the I/O activity counts for each specific system (with no data about
    aggregate disk I/Os across all systems or elapsed avg queued/waiting
    time).

    I also was pontificating that between 60s and early 80s, disk relative
    system throughput had declined by order of magnitude (disks got 3-5
    times faster while systems got 40-50 times faster). Disk division
    executive assigned the division performance group to refute the claim
    ... after a couple weeks they came back and basically I was slightly understating the problem (this got respun for customer presentations on
    how to configure disks for improved system throughput).

    --
    virtualization experience starting Jan1968, online at home since Mar1970

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Lynn Wheeler@21:1/5 to John Levine on Thu Dec 12 21:55:58 2024
    John Levine <johnl@taugh.com> writes:
    Any idea who invented CKD? The Pugh et al history book says a lot about
    disk hardware but nothing about software or CKD. Someone must have invented it but who? When?

    not who but why: https://retrocomputing.stackexchange.com/questions/24074/how-did-reserve-tracks-work-on-early-hard-disks
    Count Key Data was IBM's answer to unify three very different methods of
    data storage (Disk, Drum, Cells) into a single interface, while at the
    same time offloading basic tasks to hardware.

    discussion "spare tracks" & "standard configuration" https://archive.computerhistory.org/resources/access/text/2014/07/102739924-05-03-acc.pdf
    Jack: That was the first time in a sense that you had an index, which is now standard configuration.
    Al: Right.
    Jack: Standard configuration. The other thing that we did, that we
    subsequently cursed a lot, was the idea of variable record length and count
    key data as the format. I don't know if IBM's gotten rid of that yet.

    ... seems to imply collective "we" team/group

    and more discussion of 1301&1311 to CKD 2311 (disk) https://archive.computerhistory.org/resources/text/Oral_History/IBM_1311_2311/IBM_1311_2311.oral_history.2005.102657931.pdf

    --
    virtualization experience starting Jan1968, online at home since Mar1970

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From EricP@21:1/5 to John Levine on Fri Dec 13 14:07:32 2024
    John Levine wrote:
    According to Lynn Wheeler <lynn@garlic.com>:
    John Levine <johnl@taugh.com> writes:
    My question is where did CKD come from? I looked at the 1964 IBM Systems Journal
    issue that describes the design of S/360 which has an article describing the way
    the channels work, but nothing about the disks. Wikipedia has a long list of CKD
    drives and controllers but no hint of whose idea it was.

    One clear mistake is that they underestimated how quickly main memories would
    grow, which meant that you could cache keys in memory so most of what would >>> otherwise be found by a channel ISAM search was in the cached keys and you could
    usually directly read correct record on the disk. Similarly, if you have >>> plenty of memory to buffer multiple disk blocks, splitting logical records of
    whatever size across fixed size blocks is no big deal.

    big part of CKD was trade-off of I/O and memory. search & multi-track
    search ... have the I/O channel program search for the record you wanted
    ... at least by the mid-70s, the trade-off was inverting.
    [ long example of terrible CKD performance ]

    Any idea who invented CKD? The Pugh et al history book says a lot about
    disk hardware but nothing about software or CKD. Someone must have invented it but who? When?

    It's the 2841 Storage Control Unit that actually creates the CKD records.
    The manual referenced below says that it does CKD on a 2311 disk and
    2302 disk, but also a 2321 Data Cell Drive and a 7320 Drum drive.
    Document A26-5988-0 is the earliest reference I could find from 1965,
    but doesn't say where the idea came from.

    It references an earlier document IBM 2841 Storage Control Unit A24-3254
    but I couldn't find any copies online.

    BTW if you want to see a weird piece of storage hardware,
    take a gander at the chapter on 2321 Data Cell Drive.
    It has a rotating drum with cards in slots that move up and down
    to be read, kind of like the old slide projectors,
    and each card has 10 magnetic stripes with 200 storage locations.

    http://www.bitsavers.org/pdf/ibm/2841/

    http://www.bitsavers.org/pdf/ibm/2841/A26-5988-0_2841_2311_2321_7320_Descr.pdf

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From John Levine@21:1/5 to All on Fri Dec 13 21:45:20 2024
    According to EricP <ThatWouldBeTelling@thevillage.com>:
    Any idea who invented CKD? The Pugh et al history book says a lot about
    disk hardware but nothing about software or CKD. Someone must have invented >> it but who? When?

    It's the 2841 Storage Control Unit that actually creates the CKD records.
    The manual referenced below says that it does CKD on a 2311 disk and
    2302 disk, but also a 2321 Data Cell Drive and a 7320 Drum drive.

    Ah, I hadn't realized that. I see the 2314 had its own storage control
    and probably later CKD devices.

    BTW if you want to see a weird piece of storage hardware,
    take a gander at the chapter on 2321 Data Cell Drive.
    It has a rotating drum with cards in slots that move up and down
    to be read, kind of like the old slide projectors,
    and each card has 10 magnetic stripes with 200 storage locations.

    Never saw one but I heard they didn't work very well and had a tendency
    to crimp the tape strips when putting them away.
    --
    Regards,
    John Levine, johnl@taugh.com, Primary Perpetrator of "The Internet for Dummies",
    Please consider the environment before reading this e-mail. https://jl.ly

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Lynn Wheeler@21:1/5 to EricP on Fri Dec 13 13:03:19 2024
    EricP <ThatWouldBeTelling@thevillage.com> writes:
    It's the 2841 Storage Control Unit that actually creates the CKD records.
    The manual referenced below says that it does CKD on a 2311 disk and
    2302 disk, but also a 2321 Data Cell Drive and a 7320 Drum drive.
    Document A26-5988-0 is the earliest reference I could find from 1965,
    but doesn't say where the idea came from.

    univ got 360/67 with 2301 drum and bank of 2314 drives (replacing
    709/1401) originally for tss/360 ... never came to production so ran as
    360/65 ... when it arrived, I was hired fulltime responsible for os/360.

    univ library then got an ONR grant to do an online catalog and some of
    the money went for 2321 datacell ... also was selected as one of the
    betatest sites for original CICS transaction processing product (and
    CICS debugging/support was added to my tasks).

    when IPL'ing, 2321 made distinct sound as os/360 read the volser
    of each cell ... whirl, kerchuk, whirl, kerchuk, ....

    2303 drum ran off same controller (2821) as used for 2311s ... 2301 was
    similar to 2303 but read/wrote four tracks in parallel (four times the
    transfer rate, 1/4 the number of "tracks" and each "track" four times
    larger and needed its own controller) https://en.wikipedia.org/wiki/IBM_drum_storage https://en.wikipedia.org/wiki/History_of_IBM_CKD_Controllers

    --
    virtualization experience starting Jan1968, online at home since Mar1970

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From EricP@21:1/5 to John Levine on Sat Dec 14 11:11:26 2024
    John Levine wrote:
    According to EricP <ThatWouldBeTelling@thevillage.com>:

    BTW if you want to see a weird piece of storage hardware,
    take a gander at the chapter on 2321 Data Cell Drive.
    It has a rotating drum with cards in slots that move up and down
    to be read, kind of like the old slide projectors,
    and each card has 10 magnetic stripes with 200 storage locations.

    Never saw one but I heard they didn't work very well and had a tendency
    to crimp the tape strips when putting them away.

    There is a wikipedia on it too.

    https://en.wikipedia.org/wiki/IBM_2321_Data_Cell

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Sarr Blumson@21:1/5 to EricP on Mon Dec 23 18:40:27 2024
    EricP <ThatWouldBeTelling@thevillage.com> wrote:
    John Levine wrote:

    BTW if you want to see a weird piece of storage hardware,
    take a gander at the chapter on 2321 Data Cell Drive.
    It has a rotating drum with cards in slots that move up and down
    to be read, kind of like the old slide projectors,
    and each card has 10 magnetic stripe

    http://www.bitsavers.org/pdf/ibm/2841/

    http://www.bitsavers.org/pdf/ibm/2841/A26-5988-0_2841_2311_2321_7320_Descr.pdf

    In 1969 I was a TA for intro programming with ~300 studemts at Michigan. The Computer Center had just bought a DataCell and, because they intended it for archival storage, made it cheap. So we, being budget conscious, put the
    course files for out final problem on it. During finals week. MTBF was about
    20 minutes. Great unhappiness.

    --
    sarr@sdf.org
    SDF Public Access UNIX System - http://sdf.org

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Lars Poulsen@21:1/5 to Sarr Blumson on Mon Dec 23 21:48:11 2024
    On 2024-12-23, Sarr Blumson <sarr@sdf.org> wrote:
    In 1969 I was a TA for intro programming with ~300 studemts at Michigan. The Computer Center had just bought a DataCell and, because they intended it for archival storage, made it cheap. So we, being budget conscious, put the course files for out final problem on it. During finals week. MTBF was about 20 minutes. Great unhappiness.

    That lack of reliability is also exceptionally bad for archival storage.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)