• UK blackout, 9th August

    From johnsuth@johnsuth@nospam.com.au to aus.rail on Fri Sep 13 22:39:02 2019
    From Newsgroup: aus.rail

    Sylvia,

    I posted this in UK.rail.

    Thankyou for posting the link to the report on what happened. Here is my take on why it happened. I hope you will point out the holes in my arguments.

    Civilisation as we know it was built on coal fired steam power, slavery, deforestation, pesticides, etc..

    I note that there was no contribution from coal to the network, on the day. Readers of this forum will know that locomotive firemen would add coal to the furnace, and generate steam energy in excess of current requirement, before the train reached a hill and suddenly demanded additional power. Some of the alternatives to coal fired steam do not have this ability. The steam turbine at Little Barford employed steam generated in real time by the waste heat of the gas turbine. There was no possibility of storing steam energy in anticipation of a sudden increase in demand. Gas turbines were unsuccessful as motive power for terrestrial vehicles (cars, trucks, trains) because they are relatively (compared to steam, petrol, diesel) slow at responding to a sudden increase in power demand for hill climbing. They are no faster when their load is a network connected alternator. In addition to this fundamental weakness there were human programming flaws in the plant control system. When the steam alternator tripped, the control programmer ordered steam to be dumped directly to the condenser, and when the dump valve failed and resulted in a build up of steam pressure, there was no plan B to dump the steam directly to atmosphere, so the gas turbine alternator was tripped. Easy solution for the plant designer, painful solution for the British people.

    The tripping of the Hornsea wind generators was due to a command from the control system programmer who did not have a full understanding of the system. Easy solution for the plant designer, painful solution for the British people. South Australia went Black for the same reason, a few years before.

    The 7xx trains may have seen the same cause, or perhaps in anticipation of driverless trains, the programmer assumed there would be no driver to reboot the train in an event that was not supposed to happen anyway.

    The report told us that loss of small generators connected to the Distribution (as opposed to Transmission) system is expected in response to a lightning strike on a transmission line.

    So far the performance of substitutes for coal fired electricity suggest that we should lower our expectations of civilisation.


    --- Synchronet 3.21d-Linux NewsLink 1.2
  • From Sylvia Else@sylvia@email.invalid to aus.rail on Sat Sep 14 11:28:49 2019
    From Newsgroup: aus.rail

    On 14/09/2019 8:39 am, johnsuth@nospam.com.au wrote:


    <https://www.ofgem.gov.uk/system/files/docs/2019/09/eso_technical_report_-_final.pdf>

    The primary issue does seem to have been an unexpectedly coincidental
    trip of a generator such that the loss of generation exceeded the short
    term loss provided for (1GW).

    The outage was comparatively short, such that it is mainly the wide
    extent that made it noteworthy. We're not talking about the long
    duration widespread outages that have been seen elsewhere.

    The trains should not have behaved that way, and certainly there's
    something wrong with a system if it cannot just be shutdown and
    restarted by the driver. The reason some of the trains required a
    technician to restart then will presumably be investigated and addressed.

    Even for those trains that were rebooted by the drivers, I find it hard
    to fathom why this would take 10 minutes.

    Sylvia.
    --- Synchronet 3.21d-Linux NewsLink 1.2
  • From news18@news18@woa.com.au to aus.rail on Sat Sep 14 01:53:38 2019
    From Newsgroup: aus.rail

    On Fri, 13 Sep 2019 22:39:02 +0000, johnsuth wrote:


    The tripping of the Hornsea wind generators was due to a command from
    the control system programmer who did not have a full understanding of
    the system. Easy solution for the plant designer, painful solution for
    the British people. South Australia went Black for the same reason, a
    few years before.

    See below.

    So far the performance of substitutes for coal fired electricity suggest
    that we should lower our expectations of civilisation.

    Early days for the roll out of new tech, or rather a major new use.
    Thin back to the early days when steam power was first rolled out. They
    still had to learn that boilers needed pressure relief valves, unless you wanted them to explode off course.
    --- Synchronet 3.21d-Linux NewsLink 1.2
  • From news18@news18@woa.com.au to aus.rail on Sat Sep 14 01:55:28 2019
    From Newsgroup: aus.rail

    On Sat, 14 Sep 2019 11:28:49 +1000, Sylvia Else wrote:


    Even for those trains that were rebooted by the drivers, I find it hard
    to fathom why this would take 10 minutes.

    Safety checks. The system is booting from an unknown state; aka it has no
    idea of its operating history.

    Sylvia.

    --- Synchronet 3.21d-Linux NewsLink 1.2
  • From Sylvia Else@sylvia@email.invalid to aus.rail on Sat Sep 14 12:02:11 2019
    From Newsgroup: aus.rail

    On 14/09/2019 11:55 am, news18 wrote:
    On Sat, 14 Sep 2019 11:28:49 +1000, Sylvia Else wrote:


    Even for those trains that were rebooted by the drivers, I find it hard
    to fathom why this would take 10 minutes.

    Safety checks. The system is booting from an unknown state; aka it has no idea of its operating history.

    Yes, but this is a computer and electronics we're talking about. They
    should be able to go through all the safety checks in a fraction of that
    time.

    Sylvia.
    --- Synchronet 3.21d-Linux NewsLink 1.2
  • From Matthew Geier@matthew@sleeper.apana.org.au to aus.rail on Sat Sep 14 09:40:31 2019
    From Newsgroup: aus.rail

    On Saturday, 14 September 2019 04:02:13 UTC+2, Sylvia Else wrote:
    On 14/09/2019 11:55 am, news18 wrote:
    On Sat, 14 Sep 2019 11:28:49 +1000, Sylvia Else wrote:


    Even for those trains that were rebooted by the drivers, I find it hard
    to fathom why this would take 10 minutes.

    Safety checks. The system is booting from an unknown state; aka it has no idea of its operating history.

    Yes, but this is a computer and electronics we're talking about. They
    should be able to go through all the safety checks in a fraction of that time.
    An industrial computer network in control of real hardware. I can fully imagine a 10-minute cycle to 'power off and reboot'. It's not a desktop PC. Things are VERY different in the industrial compute world.
    I've got a $40k ethernet switch in my server room at work that takes nearly 5 minutes to boot from power on or 'reload' that does a 'cold start'. It has to load the firmware out of ROM into RAM, uncompress, then it starts scanning the bus and loads the ASIC program into multiple line cards, runs various self-tests to ensure the ASIC chips all came up properly, then brings up each card one by one, doing more checks as it goes. It takes a LONG time.
    I've been working on a 22-year-old tram. It's fully software controlled. It doesn't take 10 minutes to boot, but from battery on to being able to move, it wouldn't be much less than 10 minutes by the time you wait for the computer then do the manual startup checks required before moving off. This tram has something like 12 computers all networked - all have to start up, check the hardware they control, and report state back to the main computer. It isn't a simple process at all.
    --- Synchronet 3.21d-Linux NewsLink 1.2
  • From Petzl@petzlx@gmail.com to aus.rail on Mon Sep 16 09:53:40 2019
    From Newsgroup: aus.rail

    On Sat, 14 Sep 2019 09:40:31 -0700 (PDT), Matthew Geier <matthew@sleeper.apana.org.au> wrote:

    On Saturday, 14 September 2019 04:02:13 UTC+2, Sylvia Else wrote:
    On 14/09/2019 11:55 am, news18 wrote:
    On Sat, 14 Sep 2019 11:28:49 +1000, Sylvia Else wrote:


    Even for those trains that were rebooted by the drivers, I find it hard >> >> to fathom why this would take 10 minutes.

    Safety checks. The system is booting from an unknown state; aka it has no >> > idea of its operating history.

    Yes, but this is a computer and electronics we're talking about. They
    should be able to go through all the safety checks in a fraction of that
    time.

    An industrial computer network in control of real hardware. I can fully
    imagine a 10-minute cycle to 'power off and reboot'. It's not a desktop PC.
    Things are VERY different in the industrial compute world.

    I've got a $40k ethernet switch in my server room at work that takes
    nearly 5 minutes to boot from power on or 'reload' that does a 'cold start'. >It has to load the firmware out of ROM into RAM, uncompress,
    then it starts scanning the bus and loads the ASIC program into multiple >line cards, runs various self-tests to ensure the ASIC chips all
    came up properly, then brings up each card one by one,
    doing more checks as it goes. It takes a LONG time.


    I've been working on a 22-year-old tram. It's fully software controlled.
    It doesn't take 10 minutes to boot, but from battery on to being able to >move, it wouldn't be much less than 10 minutes by the time you wait for
    the computer then do the manual startup checks required before moving off. >This tram has something like 12 computers all networked
    - all have to start up, check the hardware they control, and report state >back to the main computer. It isn't a simple process at all.

    Would think nowadays computer controlled trains would have a
    "Uninterruptible Power Supply" or UPS.
    A short power failure would then not matter?
    --
    Petzl
    Good lawyers know the law
    Great lawyers know the judge
    --- Synchronet 3.21d-Linux NewsLink 1.2
  • From Sylvia Else@sylvia@email.invalid to aus.rail on Mon Sep 16 12:14:00 2019
    From Newsgroup: aus.rail

    On 16/09/2019 9:53 am, Petzl wrote:
    On Sat, 14 Sep 2019 09:40:31 -0700 (PDT), Matthew Geier <matthew@sleeper.apana.org.au> wrote:

    On Saturday, 14 September 2019 04:02:13 UTC+2, Sylvia Else wrote:
    On 14/09/2019 11:55 am, news18 wrote:
    On Sat, 14 Sep 2019 11:28:49 +1000, Sylvia Else wrote:


    Even for those trains that were rebooted by the drivers, I find it hard >>>>> to fathom why this would take 10 minutes.

    Safety checks. The system is booting from an unknown state; aka it has no >>>> idea of its operating history.

    Yes, but this is a computer and electronics we're talking about. They
    should be able to go through all the safety checks in a fraction of that >>> time.

    An industrial computer network in control of real hardware. I can fully
    imagine a 10-minute cycle to 'power off and reboot'. It's not a desktop PC. >> Things are VERY different in the industrial compute world.

    I've got a $40k ethernet switch in my server room at work that takes
    nearly 5 minutes to boot from power on or 'reload' that does a 'cold start'. >> It has to load the firmware out of ROM into RAM, uncompress,
    then it starts scanning the bus and loads the ASIC program into multiple
    line cards, runs various self-tests to ensure the ASIC chips all
    came up properly, then brings up each card one by one,
    doing more checks as it goes. It takes a LONG time.


    I've been working on a 22-year-old tram. It's fully software controlled.
    It doesn't take 10 minutes to boot, but from battery on to being able to
    move, it wouldn't be much less than 10 minutes by the time you wait for
    the computer then do the manual startup checks required before moving off. >> This tram has something like 12 computers all networked
    - all have to start up, check the hardware they control, and report state
    back to the main computer. It isn't a simple process at all.

    Would think nowadays computer controlled trains would have a
    "Uninterruptible Power Supply" or UPS.
    A short power failure would then not matter?


    Some of the trains appear to have choked on the frequency being too far
    from nominal (which they were not meant to), rather than because of the
    outage itself.

    Sylvia.
    --- Synchronet 3.21d-Linux NewsLink 1.2
  • From Matthew Geier@matthew@sleeper.apana.org.au to aus.rail on Sun Sep 15 21:40:31 2019
    From Newsgroup: aus.rail

    On Monday, 16 September 2019 01:53:46 UTC+2, Petzl wrote:

    Would think nowadays computer controlled trains would have a
    "Uninterruptible Power Supply" or UPS.
    A short power failure would then not matter?

    It wasn't the computer failing, the computer detected an 'out of tollerence' situation with the incoming power supply and shut the train down to 'protect' the equipment.
    But instead of resuming normal operation when the problem when away, they 'hard tripped' the traction converters.
    The documented restart procedure was to shutdown and restart (stable and then prep). This takes 10 minutes.
    However, 30 train sets had software updates that removed this particular lockout from being reset by a shutdown.
    None of this behaviour was actually within specification. The trains had been accepted into service with software that acted differently from the specification.
    I've actually got a rollingstock control computer stuck in a fault condition that could only be resolved by disconnecting the NVRAM battery (with a soldering iron!). What we think is a bug in the 25-year-old diagnostic software, meant it couldn't reset a flag in the data logger in the state we had gotten the system into.
    And no support from the vendor either :-)
    --- Synchronet 3.21d-Linux NewsLink 1.2
  • From news18@news18@woa.com.au to aus.rail on Mon Sep 16 08:37:25 2019
    From Newsgroup: aus.rail

    On Sat, 14 Sep 2019 12:02:11 +1000, Sylvia Else wrote:

    On 14/09/2019 11:55 am, news18 wrote:
    On Sat, 14 Sep 2019 11:28:49 +1000, Sylvia Else wrote:


    Even for those trains that were rebooted by the drivers, I find it
    hard to fathom why this would take 10 minutes.

    Safety checks. The system is booting from an unknown state; aka it has
    no idea of its operating history.

    Yes, but this is a computer and electronics we're talking about. They
    should be able to go through all the safety checks in a fraction of that time.

    No, you are talkng about a lot of inductive stuff, like motors. The logic might be able to run throught its programm in u-seconds, but in a real
    world there are real compnents to be measured.

    Simpple stuff like disengaging the drive before powering a moter and
    checking that it responds "normally".

    Then there are thermal checks to be run It is going to want to do
    somethigk to indicate if it can at least crawl to a siding or does it
    require a tow from the gitgo.
    --- Synchronet 3.21d-Linux NewsLink 1.2
  • From Sylvia Else@sylvia@email.invalid to aus.rail on Mon Sep 16 20:05:06 2019
    From Newsgroup: aus.rail

    On 16/09/2019 6:37 pm, news18 wrote:
    On Sat, 14 Sep 2019 12:02:11 +1000, Sylvia Else wrote:

    On 14/09/2019 11:55 am, news18 wrote:
    On Sat, 14 Sep 2019 11:28:49 +1000, Sylvia Else wrote:


    Even for those trains that were rebooted by the drivers, I find it
    hard to fathom why this would take 10 minutes.

    Safety checks. The system is booting from an unknown state; aka it has
    no idea of its operating history.

    Yes, but this is a computer and electronics we're talking about. They
    should be able to go through all the safety checks in a fraction of that
    time.

    No, you are talkng about a lot of inductive stuff, like motors. The logic might be able to run throught its programm in u-seconds, but in a real
    world there are real compnents to be measured.

    Simpple stuff like disengaging the drive before powering a moter and checking that it responds "normally".

    Should take seconds at most.


    Then there are thermal checks to be run It is going to want to do
    somethigk to indicate if it can at least crawl to a siding or does it
    require a tow from the gitgo.

    There's only so much that can be done without applying traction current,
    and those parts that handle traction current are not going to heat up measurably otherwise.

    I've seen the way people work. It's probably a long sequence of
    unreasonably high timeouts. They didn't know what was a sensible value
    to use, and rather than try to find out, they just stuck in something
    that was bound to be enough.

    Sylvia.
    --- Synchronet 3.21d-Linux NewsLink 1.2
  • From Matthew Geier@matthew@sleeper.apana.org.au to aus.rail on Mon Sep 16 08:45:25 2019
    From Newsgroup: aus.rail

    On Monday, 16 September 2019 12:05:10 UTC+2, Sylvia Else wrote:

    Should take seconds at most.
    The train computer I've been working on waits 5 seconds from the application of power to even begin booting. This is hard-wired into the power supply. The reason is to ensure the incoming supply is stable before starting to boot. It also has to boot within 30 seconds, otherwise, it can't supply a keep-alive and the power control contactor drops out. But that's just the start of the main program loop. That doesn't mean the train is ready.
    Part of said computer's initialisation is to assign addresses to devices on the 'train bus', which is (by today's networking standards) a slow speed serial bus. It's specially designed to be robust, not fast. Depending on the number of devices on this bus, just that search and initialization could take minutes.
    This is not a desktop PC. Near enough is not good enough.
    --- Synchronet 3.21d-Linux NewsLink 1.2
  • From johnsuth@johnsuth@nospam.com.au to aus.rail on Sun Sep 15 01:18:07 2019
    From Newsgroup: aus.rail

    In <qlhh72$cga$5@dont-email.me>, news18 <news18@woa.com.au> writes:
    On Fri, 13 Sep 2019 22:39:02 +0000, johnsuth wrote:


    The tripping of the Hornsea wind generators was due to a command from
    the control system programmer who did not have a full understanding of
    the system. Easy solution for the plant designer, painful solution for
    the British people. South Australia went Black for the same reason, a
    few years before.

    See below.

    So far the performance of substitutes for coal fired electricity suggest
    that we should lower our expectations of civilisation.

    Early days for the roll out of new tech, or rather a major new use.
    Thin back to the early days when steam power was first rolled out. They >still had to learn that boilers needed pressure relief valves, unless you >wanted them to explode off course.

    We have put a man on the Moon since then.


    --- Synchronet 3.21d-Linux NewsLink 1.2
  • From news18@news18@woa.com.au to aus.rail on Sat Sep 21 13:07:52 2019
    From Newsgroup: aus.rail

    On Mon, 16 Sep 2019 20:05:06 +1000, Sylvia Else wrote:

    On 16/09/2019 6:37 pm, news18 wrote:
    On Sat, 14 Sep 2019 12:02:11 +1000, Sylvia Else wrote:

    On 14/09/2019 11:55 am, news18 wrote:
    On Sat, 14 Sep 2019 11:28:49 +1000, Sylvia Else wrote:


    Even for those trains that were rebooted by the drivers, I find it
    hard to fathom why this would take 10 minutes.

    Safety checks. The system is booting from an unknown state; aka it
    has no idea of its operating history.

    Yes, but this is a computer and electronics we're talking about. They
    should be able to go through all the safety checks in a fraction of
    that time.

    No, you are talkng about a lot of inductive stuff, like motors. The
    logic might be able to run throught its programm in u-seconds, but in a
    real world there are real compnents to be measured.

    Simpple stuff like disengaging the drive before powering a moter and
    checking that it responds "normally".

    Should take seconds at most.


    Then there are thermal checks to be run It is going to want to do
    somethigk to indicate if it can at least crawl to a siding or does it
    require a tow from the gitgo.

    There's only so much that can be done without applying traction current,
    and those parts that handle traction current are not going to heat up measurably otherwise.

    I've seen the way people work. It's probably a long sequence of
    unreasonably high timeouts. They didn't know what was a sensible value
    to use, and rather than try to find out, they just stuck in something
    that was bound to be enough.

    Naah, they have to cater for all sorts of possible failures and want tne minimal possible power-on damage. It cheaper to replace a minore item
    thana to let major smoke out.

    Sylvia.

    --- Synchronet 3.21d-Linux NewsLink 1.2
  • From news18@news18@woa.com.au to aus.rail on Sat Sep 21 13:09:38 2019
    From Newsgroup: aus.rail

    On Sun, 15 Sep 2019 01:18:07 +0000, johnsuth wrote:

    In <qlhh72$cga$5@dont-email.me>, news18 <news18@woa.com.au> writes:
    On Fri, 13 Sep 2019 22:39:02 +0000, johnsuth wrote:


    The tripping of the Hornsea wind generators was due to a command from
    the control system programmer who did not have a full understanding of
    the system. Easy solution for the plant designer, painful solution
    for the British people. South Australia went Black for the same
    reason, a few years before.

    See below.

    So far the performance of substitutes for coal fired electricity
    suggest that we should lower our expectations of civilisation.

    Early days for the roll out of new tech, or rather a major new use.
    Thin back to the early days when steam power was first rolled out. They >>still had to learn that boilers needed pressure relief valves, unless
    you wanted them to explode off course.

    We have put a man on the Moon since then.

    Not many and the death toll is noticeable.

    --- Synchronet 3.21d-Linux NewsLink 1.2