Description
I am concerned about the communication stability of my BACnet network. How much load will a Coppercube add to my BACnet network?
Solution
Obviously, the amount of bandwidth used is a function of the number of trend logs, & objects the Coppercube is collecting and the number of controller databases it discovers needing to be backed up. All of which is a coarse reflection of site size. The bigger the site, the more data that must be collected, and the larger the impact.
For starters, let’s examine some basic knowledge about BACnet, the Coppercube, and Delta Controls controllers (DAC/DSC specifically).
- the Delta DAC/DSC family of controllers uses 480-byte packets when communicating over MSTP or BACnet/Ethernet.
- a single 480-byte BACnet packet holds 21 trend log samples (i.e. one time-value-sequence tuple).
- a Delta MTSP network segment can have 99 controllers, but 50 controllers are the practical limit. [BACnet can have 127 Masters, & 127 Slaves – which we will ignore as they complicate an already complicated discussion].
- let’s assume an MSTP controller (VAV, Fancoil, etc) has a typical load of 10 TLs.
- by default, a Coppercube visits each TL every 4 hrs.
- the Coppercube supports collecting COV TLs & Polled TLs with up to once-a-minute sampling rates [controllers are capable of even faster sampling rates].
- TLs of interest in analytics have typical sampling rates of 5 min, 15 min, or hourly.
- Let’s assume a 5 min sampling rate for all TLs.
- COV TLs are non-deterministic. Only polled TLs will be considered here-in.
The questions then are:
- How many TLs are being collected? How many samples per day?
- How many BACnet Objects are being collected? How big is each one?
- How many controller backups occur each day?
Trend log collection
Given our assumptions (above), let’s examine a fully loaded Coppercube (5000 TLs sampling at 5 min). We know:
- a Polled TL sampling at 5 min collects: 12 samples / hr = 12 * 4 = 48 samples within the Coppercube 4 hr visit interval.
- at 21 samples per BACnet packet; it takes 48 / 21 = 3 packets to extract the accumulated samples.
- at 6 visits/day, it takes 6 x 3 packets = 18 * 480 bytes = 8640 bytes / 1024 = 8.5K per TL per day.
- with 5000 TLs, it takes 5000 x 18 packets = 90000 packets OR 5000 x 8.5K = 42500K / 1024 = 41.5MB/day
But how does that compare to MSTP’s available bandwidth?
- Delta Controls uses 76.8K MSTP. Meaning 76800 baud / 10bits = 7680 bytes / 480 byte packets = 16 packets/sec of available bandwidth.
- MSTP has ~50% overhead (token passing), leaving 8 packets/sec for useful payloads.
- at 3 packets / TL visit = 3 / 8 = 0.375 sec to visit a TL. [or 2.6 TLs/sec]
If we naively compute (5000 TLs * 0.375 sec = 1875 sec / 60 = 31.3 min / 4 hr window) we’d get 13% of MSTP is consumed collecting TL data. Of course, it is not that simple. Firstly, a single segment MSTP network of sufficient size to max a Coppercube cannot be constructed in practice. Why?
Because, MSTP addressing prevents constructing a network with 5000 controllers (one TL each) or even 500 controllers (100 TLs each), and device performance limitations typically make segments of 100 devices (50 TLs each) unreliable. A more realistic MSTP configuration would be 10 MSTP segments of 50 devices (10 TLs each). The happy consequence of this ‘good MSTP architecture’ is the load is automatically spread across the MSTP segments by the nature of MSTP routing causing the TL data to (generally) be collected in parallel.
Let’s assume these segments are joined with Ethernet. For if they are were joined with an MSTP backbone (an obsolete architecture) we’d find the network would become unusable under the load. But wait, you say, that is my situation! How bad will it be?! [see Bad Situations]. With an Ethernet backbone in place, we can safely ignore the load on the Ethernet portion (due to its much greater bandwidth) and focus on the load imposed on a single MSTP segment. Recall the load will be spread across the 10 segments, so each one will see only 1/10 of the total load.
- Assume the MSTP segment has 50 controllers with 10 TLs each = 500 TLs * 0.375 sec = 187.5 sec / 60 = 3.1 mins to collect those TLs every 4 hrs.
- meaning 188 sec spread over 4 hrs = 188 / (4 * 3600) = 0.013 * 100 = 1.3% of the segment’s bandwidth is consumed by TL collection.
- Because every MSTP segment is exposed to the same load, we can state the entire MSTP network experiences a load of 1.3% of bandwidth due to TL collection.
- we could account for the data relying on done by the MSTP-to-Ethernet routing controllers (DSCs) – but since Ethernet (even 10Mbit) has so much more bandwidth the percentage consumed is not significant. The percentage on Ethernet (10Mbit / 10bits = 1MB/sec vs. 7680 bytes/sec, is (1.3% / 136x more bandwidth) is ~0.009%
BACnet Object collection
BACnet Object collection is another significant load. Individual objects are highly variable in size and quantity, and very difficult to estimate. The rule of thumb is typically 10 objects per TL. If a controller has 50 TLs, it will have 500 Objects to accompany these TLs. Let’s assume object collection imposes a load equal to TL collection – i.e. another 1.3% on MSTP. [KbA0010 shows this a reasonable assumption.]
Database Backups
By default, controller databases are backed up once a week. A typical DAC database is 200K. Since this is not a daily item, let’s exclude this traffic from the discussion. But we will keep in mind that once-a-week Database collection will impose an additional load equal to object collection upon the network (i.e. 1.3%). While the databases actually contain all the objects and much of the TL data held by controllers, they are transmitted in a compact binary format much smaller than the standard BACnet data exchange format. As a result, the total data quantity is roughly the same as just collecting the objects.
By adding the TL data volume to the BACnet Object data volume, (given our network assumptions) we can state a maximum loaded Coppercube:
- will generate 41.5MB/day * 2 = 83MB/day of MSTP data spread across all the MSTP segments.
- will consume 3% of the MSTP bandwidth. A small (but noticeable) amount of traffic.
- and one-day-a-week (usually Sunday) it will consume 4.3% of the MSTP bandwidth.
Bad Situations
MSTP Backbone
An MSTP network constructed with an MSTP backbone is a recipe-for-disaster. The backbone segment will see all the traffic load of all the subnet segments plus its own peer-to-peer traffic. Recall in our assumed 10-segment MSTP network, we stated each segment will see 1.3% of bandwidth consumed with TL collection and another 1.3% consumed with object collection. Since the backbone sees all the traffic of all the subnets; the backbone will immediately see (1.3% x 2)x 10 = 26% of its bandwidth consumed. “Not great but still survivable”- you say. Not so fast. How does MSTP behave under increasing loads?MSTP is a token-passing network. That means it ensures every device gets a chance to transmit its data. The more devices on the network, the longer the interval between when each device may transmit, but they will get to transmit. The cost of providing this guarantee is the large overhead of passing the token (up to 50% of available bandwidth). This is opposed to Ethernet which is collision-based – meaning the overhead is minimal, but there is no guarantee a device will be able to transmit its data.
In both designs, throughput decreases as load increases – until some tipping point – and then things “go-to-hell”. Ethernet is well studied. Its tipping point is known to be 30% of capacity – at which point packet collisions & retries cause the network to thrash. With MSTP collisions don’t occur, data just gets delayed (but still gets through). Its tipping point is higher (70%-80% according to NISTIR 7038: A Simulation Analysis of BACnet) but once crossed, it degrades exponentially. Also beware that when the delay approaches various (application) timeout values (5-7 sec), the generation of retries adds even more traffic, leading to further delays; causing devices to be ‘marked offline’; creating ‘Who-Is’/’I-am’ storms; etc. Just like Ethernet, once the tipping point is reached, the network thrashes.
Field experience suggests it is not difficult to reach the tipping point of MSTP. There is a lot of traffic in a typical network to users are unaware of. The biggest traffic generators are operator workstations, followed by network housekeeping (Who-Is/I-Am/Who-Has), data exchange, and alarms. Add to this significant existing baseload, another 26%. Assume the network is using 50% of available bandwidth in its routine operations. Another 26% will push it past the tipping point. Any doubt left why – MSTP backbones are a recipe-for-disaster? They simply don’t have sufficient bandwidth.
38400 baud MSTP
All the above calculations assumed 78600 baud MSTP. What happens if you are using 38400 baud? Firstly, your available bandwidth is cut in half. But the amount of data and the data collection frequency has not gone down. Same load — 1/2 the bandwidth = twice the bandwidth consumption. As best this means bandwidth consumption becomes (1.3% * 2)TL + (1.3% * 2)Object = 5.2% on each segment. 5.2% + 2.6% = 7.8% during database collection (Sunday).In practice, a slower MSTP baud rate, demands reducing the amount of data you collect.
