The Telecom Site That Was Slowly Sinking

This Tales From the Field article was contributed by Daniel Cáceres, PMP

What a remote network outage revealed about infrastructure monitoring, residual risk, and the physical conditions supporting a telecom asset

A Routine Alarm

Some of the most important lessons in infrastructure begin with events that initially look routine.

During my work managing telecommunications infrastructure and operations in Peru, a Network Operations Center detected that the optical equipment at a remote telecommunications node had gone offline. When that equipment stopped operating, the links dependent on the node—and traffic passing through it—were affected.

The NOC generated an incident ticket and transferred it to the maintenance contractor responsible for field response.

At that moment, the contractor’s crew was completing another fault intervention some distance away. This was in a rural area of the Peruvian highlands, where travel between sites can involve long distances, mountainous terrain and limited road accessibility.

Before responding to the new incident, the crew had to return to its base, collect the necessary materials and organize the personnel required for the intervention.

The field team consisted of two operational specialties. Outside-plant personnel were prepared to respond if the problem involved the fiber-optic network, including the possibility of a cable cut or section replacement. Inside-plant personnel were responsible for power systems, telecommunications equipment and related technical systems.

It was a normal configuration for an operational maintenance response.

What it did not include was a civil works or infrastructure team.

There was no reason, at that point, to expect that one would be needed.

The service-level agreement established an eight-hour restoration window. After returning to base and preparing for the intervention, the crew began the journey toward the affected node.

It took approximately four hours to reach the site.

They arrived close to 5:00 p.m., with daylight already becoming limited.

That was when a network outage became something else.

The Door Would Not Open

When the crew attempted to enter the site, they discovered that the shelter door could not be opened normally.

Material and mud around the entrance were cleared sufficiently to inspect the condition. The relative settlement around the entrance area was approximately 10 to 15 centimeters.

The site appeared to have experienced physical settlement.

The team had traveled expecting to troubleshoot fiber, power or telecommunications equipment. Instead, before the normal intervention could even begin, they faced a physical infrastructure condition for which an operational telecom crew was not configured.

The situation was escalated to Operations management in Lima.

Because the node had lost power, normal site lighting and other electrically supplied systems were unavailable. Before authorizing an extraordinary entry, the conditions around the site—including the perimeter security elements—had to be considered so the crew could access the installation without creating an additional safety risk.

After the situation was assessed, extraordinary access was authorized.

Using a telescopic ladder, the technicians entered the site and gained access to the equipment and generator areas.

Only then could the original network incident be properly investigated.

The Network Problem Was an Energy Problem

The inside-plant technicians began their checks.

Initial checks did not identify the optical equipment itself as the immediate source of the outage. The more immediate problem was simpler: it was not receiving power.

The crew had brought fuel as a contingency for the standby generator. They supplied the generator, started it and restored electrical power to the node.

The telecommunications equipment came back online.

The NOC verified service recovery and the affected network links returned to operation.

Approximately seven to seven and a quarter hours after the incident-response process began, service had been restored—within the eight-hour SLA.

From the network perspective, the immediate objective had been achieved.

Alarm detected.
Ticket generated.
Crew dispatched.
Fault diagnosed.
Power restored.
Network recovered.
Ticket closed.

But the condition of the site was far from closed.

For the NOC, the outage had been resolved. For the infrastructure team, another problem had just begun.

Restoring the Node Did Not Restore the Site

Once the generator was operating, the team could assess the energy system more completely.

Field measurements showed that the battery bank had reached a deeply discharged condition. With the generator running, the rectifier and battery-control system indicated that charging was being applied.

But the batteries were not recovering.

Even after several hours with the charging system operating, the bank failed to regain charge as expected.

The operational evidence therefore indicated that the battery bank had become degraded and could no longer be relied upon to provide the required backup function.

A complete replacement bank was requested from the contractor’s base for the following morning.

The sequence available to the operations team suggested that the batteries may have continued supporting the site after generator availability was lost, potentially contributing to the deep-discharge condition. However, the available evidence did not justify attributing the degradation to one single cause.

This distinction is important in field analysis.

What we observe, what the equipment records and what we infer from the sequence of events are related—but they are not always the same thing.

The immediate operational priority remained maintaining the node in service.

Personnel continued supporting the site and maintaining generator operation while the next intervention was organized.

At approximately 9:00 a.m. the following day, another crew arrived with a replacement battery bank. With the new batteries installed, the node could be sustained through the available combination of local electrical supply, battery backup and standby generation as conditions required.

The telecommunications service was operating again.

The site, however, was still physically settled.

A Second Team for a Different Problem

The original field crew had done what it had been deployed to do: diagnose and restore telecommunications service.

Understanding the physical condition required different expertise.

Civil works and infrastructure personnel were mobilized to the site and arrived around midday the following day.

Initial field investigations began, including soil-related assessments and exploratory work intended to evaluate possible subsurface and environmental conditions associated with the settlement.

The objective was not to choose a convenient explanation.

It was to begin eliminating possibilities and building evidence.

Ground conditions, moisture, soil behavior and possible subsurface movement were among the conditions requiring technical assessment.

At that stage, the visible settlement was a fact.

Its precise cause was a question.

That distinction matters.

Finding a condition that appears technically plausible does not automatically establish causality. Several factors may interact over time, and the evidence required to demonstrate a mechanism can be different from the evidence required simply to observe its effects.

Professional experience helps us know where to look.

Evidence determines what we can responsibly say.

What the Monitoring System Could See

The incident also exposed a different kind of gap.

Remote telecommunications sites can generate substantial operational information.

Network management systems can detect equipment outages and communication failures. Power systems can report alarms and operating states. Battery and rectifier controllers provide energy-related information. Security systems and video surveillance can provide another layer of visibility.

All of those capabilities are valuable.

But a monitoring system only observes the variables it has been designed and instrumented to observe.

The NOC knew that the node was offline.

The operational systems could help technicians understand the state of the telecommunications and energy equipment.

What those systems had not been designed to measure was the progressive physical settlement of the site itself.

There was no dedicated variable telling Operations:

the ground around this infrastructure is moving.

That became one of the most important lessons from the event:

Monitoring whether an asset is operating is not the same as monitoring the physical conditions supporting that asset.

The monitoring system knew when the site stopped working.

It could not tell us that the ground beneath it had been slowly moving.

Turning an Existing Camera into a Temporary Monitoring Tool

While the permanent infrastructure condition required further assessment and corrective action, Operations still needed a practical way to observe whether the visible settlement was continuing.

The response did not begin with a sophisticated new sensor.

It began with something much simpler.

A graduated visual reference was placed at the affected area. A scale of approximately 20 to 30 centimeters was marked so that relative changes could be visually compared over time.

The site’s existing surveillance camera could then be positioned and zoomed toward that reference.

Instead of providing only security visibility, the camera now supported periodic observation of the physical condition.

Checks could be made at different times during the day—at the beginning of the day, around midday and later in the day—to identify whether the visible reference was changing.

It was a practical field control using infrastructure that already existed.

It provided additional information.

But it was not a repair.

Monitoring a risk is not the same as treating the risk.

The reference could help show whether movement continued.

It could support escalation.

It could preserve evidence.

It could not stabilize the site.

That difference is easy to overlook. Once a condition is visible on a screen, recorded in a report or included in an inspection routine, there is a temptation to feel that it is under control.

Sometimes it is.

Sometimes we have simply become better at watching it.

One Incident, Three Layers

What began as one NOC ticket ultimately revealed three different layers of the same asset.

The first was the network layer.

The optical equipment had gone offline, affecting links and traffic through the node.

The second was the energy layer.

The equipment was not receiving the power required to operate, and the battery bank was found in a deeply discharged and degraded condition that required replacement.

The third was the physical infrastructure layer.

The site had experienced settlement significant enough to prevent normal access and require civil and geotechnical assessment.

Each layer required a different response.

Network service had to be restored.

Energy continuity had to be re-established.

The physical condition had to be investigated, monitored and followed by appropriate corrective action.

This is why a telecommunications asset cannot always be understood only from the network equipment installed inside it.

The equipment depends on energy. The energy systems depend on physical infrastructure.

And the physical infrastructure depends on the environment in which the asset was built.

The layers are organizationally separable. Operationally, they are one system.

The SLA Was Met. The Problem Was Not Over.

There is another lesson in the timing of this incident.

The service was restored within the established eight-hour SLA.

That mattered.

For customers and network operations, restoring connectivity was the immediate priority, and the maintenance response achieved that objective.

But the event also demonstrated the boundary of an operational performance indicator.

An SLA can tell us whether service was restored within an agreed period.

It does not necessarily tell us whether every condition threatening the asset has been eliminated.

A service can be restored within SLA while the condition that threatens the asset remains unresolved.

There is no contradiction in those two statements.

They measure different things. The same applies to closing an incident ticket.

The NOC was correct to close the network outage once service had been verified as restored.

The physical infrastructure problem simply required a different process, different expertise and a different time horizon.

Restoring service and restoring infrastructure are not always the same thing.

The Project Was Already Over

By the time this event occurred, the node was not a construction project.

It was an operating telecommunications asset.

Operations and Maintenance was responsible for keeping the service available.

Yet the condition discovered at the site crossed the boundaries between telecommunications operations, energy systems and civil infrastructure.

This is where lifecycle thinking becomes important.

Infrastructure projects commonly move through stages such as:

Feasibility → Engineering → Construction → Commissioning → Handover → Operations

Each stage has its own specialists, responsibilities, deliverables and acceptance criteria.

The asset does not experience those organizational boundaries.

It experiences their accumulated result.

Once handed over, Operations receives the infrastructure as it actually exists—not as separate engineering drawings, construction packages, commissioning protocols or contractual workstreams.

That leads to a principle I have increasingly considered important:

Every project stage may have its own deliverables, but there is only one final asset.

When an unexpected physical condition emerges during operation, responsibility can become more complex than the original maintenance ticket.

Who evaluates it?

Who owns the residual risk?

Who has authority to define a permanent solution?

Who has the appropriate technical expertise?

And who can authorize and fund an intervention that may extend beyond routine O&M?

The people who discover a problem are not always the people with the authority or resources required to permanently resolve it.

That is one reason the transition from project delivery to operational ownership deserves more attention than a simple transfer of equipment and documentation.

What O&M Really Inherits

Operations is often where infrastructure conditions finally become visible because Operations is where an asset spends most of its life.

That does not mean every operational problem originated during project implementation.

Nor does it mean every future condition could reasonably have been predicted.

But handover should be understood as more than transferring an operational asset.

O&M may also receive maintenance requirements, environmental exposure, operating constraints and residual risks.

Some will already be known.

Others may only become visible after sustained operation.

This makes one feedback loop particularly important:

PROJECT → ASSET → OPERATIONS

should eventually be complemented by:

OPERATIONS → EVIDENCE → LESSONS → FUTURE PROJECTS

Without that return path, valuable field experience can remain trapped inside incident reports, maintenance tickets or the memories of the people who responded.

And another project may eventually encounter the same lesson as though it were new.

What Stayed With Me

The original event was a telecommunications outage.

The team restored the service.

The battery bank was replaced.

Civil and infrastructure specialists began investigating the physical condition.

A simple visual reference helped provide additional monitoring while the site remained operational.

But the most valuable result was the lesson left behind.

The incident reinforced for me that operational monitoring is not the same as complete asset awareness.

That restoring service does not necessarily mean restoring the infrastructure condition associated with the event.

That evidence should come before causal conclusions.

That monitoring a risk does not mean the risk has been treated.

And that the operational history of an asset should eventually become input for the next generation of projects.

The experience also changed the way I think about responsibility across the infrastructure lifecycle.

Different stakeholders can own different deliverables.

Different teams can successfully complete their assigned tasks.

But someone must maintain visibility of the integrity of the integrated outcome.

Because years after commissioning, when an alarm appears and a field crew travels several hours through difficult terrain to reach a remote site, the asset does not present itself as separate project phases.

It presents itself as one reality.

O&M receives the failure. But it also receives the reality.

Author

Daniel Cáceres, PMP is an Electronic Engineer and senior project management professional with experience in telecommunications and digital infrastructure in Peru. His professional work has included fiber-optic networks, telecommunications infrastructure, project implementation, operations and maintenance, risk management, and infrastructure lifecycle management.

Tales From the Field tells stories from the infrastructure front lines that we can learn from or laugh at.  If you have a story you’d like to tell, contact us!

If you haven't already, please take our Reader Survey! Just 3 questions to help us better understand who is reading Telecom Ramblings so we can serve you better!