Login
The U.S. government is warning that “smart locks” securing entry to an estimated 50,000 dwellings nationwide contain hard-coded credentials that can be used to remotely open any of the locks. The lock’s maker Chirp Systems remains unresponsive, even though it was first notified about the critical weakness in March 2021. Meanwhile, Chirp’s parent company, RealPage, Inc., is being sued by multiple U.S. states for allegedly colluding with landlords to illegally raise rents.
On March 7, 2024, the U.S. Cybersecurity & Infrastructure Security Agency (CISA) warned about a remotely exploitable vulnerability with “low attack complexity” in Chirp Systems smart locks.
“Chirp Access improperly stores credentials within its source code, potentially exposing sensitive information to unauthorized access,” CISA’s alert warned, assigning the bug a CVSS (badness) rating of 9.1 (out of a possible 10). “Chirp Systems has not responded to requests to work with CISA to mitigate this vulnerability.”
Matt Brown, the researcher CISA credits with reporting the flaw, is a senior systems development engineer at Amazon Web Services. Brown said he discovered the weakness and reported it to Chirp in March 2021, after the company that manages his apartment building started using Chirp smart locks and told everyone to install Chirp’s app to get in and out of their apartments.
“I use Android, which has a pretty simple workflow for downloading and decompiling the APK apps,” Brown told KrebsOnSecurity. “Given that I am pretty picky about what I trust on my devices, I downloaded Chirp and after decompiling, found that they were storing passwords and private key strings in a file.”
Using those hard-coded credentials, Brown found an attacker could then connect to an application programming interface (API) that Chirp uses which is managed by smart lock vendor August.com, and use that to enumerate and remotely lock or unlock any door in any building that uses the technology.
Update, April 18, 11:55 a.m. ET: August has provided a statement saying it does not believe August or Yale locks are vulnerable to the hack described by Brown.
“We were recently made aware of a vulnerability disclosure regarding access control systems provided by Chirp, using August and Yale locks in multifamily housing,” the company said. “Upon learning of these reports, we immediately and thoroughly investigated these claims. Our investigation found no evidence that would substantiate the vulnerability claims in either our product or Chirp’s as it relates to our systems.”
Brown said when he complained to his leasing office, they sold him a small $50 key fob that uses Near-Field Communications (NFC) to toggle the lock when he brings the fob close to his front door. But he said the fob doesn’t eliminate the ability for anyone to remotely unlock his front door using the exposed credentials and the Chirp mobile app.
Also, the fobs pass the credentials to his front door over the air in plain text, meaning someone could clone the fob just by bumping against him with a smartphone app made to read and write NFC tags.
Neither August nor Chirp Systems responded to requests for comment. It’s unclear exactly how many apartments and other residences are using the vulnerable Chirp locks, but multiple articles about the company from 2020 state that approximately 50,000 units use Chirp smart locks with August’s API.
Roughly a year before Brown reported the flaw to Chirp Systems, the company was bought by RealPage, a firm founded in 1998 as a developer of multifamily property management and data analytics software. In 2021, RealPage was acquired by the private equity giant Thoma Bravo.
Brown said the exposure he found in Chirp’s products is “an obvious flaw that is super easy to fix.”
“It’s just a matter of them being motivated to do it,” he said. “But they’re part of a private equity company now, so they’re not answerable to anybody. It’s too bad, because it’s not like residents of [the affected] properties have another choice. It’s either agree to use the app or move.”
In October 2022, an investigation by ProPublica examined RealPage’s dominance in the rent-setting software market, and that it found “uses a mysterious algorithm to help landlords push the highest possible rents on tenants.”
“For tenants, the system upends the practice of negotiating with apartment building staff,” ProPublica found. “RealPage discourages bargaining with renters and has even recommended that landlords in some cases accept a lower occupancy rate in order to raise rents and make more money. One of the algorithm’s developers told ProPublica that leasing agents had ‘too much empathy’ compared to computer generated pricing.”
Last year, the U.S. Department of Justice threw its weight behind a massive lawsuit filed by dozens of tenants who are accusing the $9 billion apartment software company of helping landlords collude to inflate rents.
In February 2024, attorneys general for Arizona and the District of Columbia sued RealPage, alleging RealPage’s software helped create a rental monopoly.
The U.S. Cybersecurity and Infrastructure Security Agency (CISA) said today it is investigating a breach at business intelligence company Sisense, whose products are designed to allow companies to view the status of multiple third-party online services in a single dashboard. CISA urged all Sisense customers to reset any credentials and secrets that may have been shared with the company, which is the same advice Sisense gave to its customers Wednesday evening.
New York City based Sisense has more than a thousand customers across a range of industry verticals, including financial services, telecommunications, healthcare and higher education. On April 10, Sisense Chief Information Security Officer Sangram Dash told customers the company had been made aware of reports that “certain Sisense company information may have been made available on what we have been advised is a restricted access server (not generally available on the internet.)”
“We are taking this matter seriously and promptly commenced an investigation,” Dash continued. “We engaged industry-leading experts to assist us with the investigation. This matter has not resulted in an interruption to our business operations. Out of an abundance of caution, and while we continue to investigate, we urge you to promptly rotate any credentials that you use within your Sisense application.”
In its alert, CISA said it was working with private industry partners to respond to a recent compromise discovered by independent security researchers involving Sisense.
“CISA is taking an active role in collaborating with private industry partners to respond to this incident, especially as it relates to impacted critical infrastructure sector organizations,” the sparse alert reads. “We will provide updates as more information becomes available.”
Sisense declined to comment when asked about the veracity of information shared by two trusted sources with close knowledge of the breach investigation. Those sources said the breach appears to have started when the attackers somehow gained access to the company’s Gitlab code repository, and in that repository was a token or credential that gave the bad guys access to Sisense’s Amazon S3 buckets in the cloud.
Customers can use Gitlab either as a solution that is hosted in the cloud at Gitlab.com, or as a self-managed deployment. KrebsOnSecurity understands that Sisense was using the self-managed version of Gitlab.
Both sources said the attackers used the S3 access to copy and exfiltrate several terabytes worth of Sisense customer data, which apparently included millions of access tokens, email account passwords, and even SSL certificates.
The incident raises questions about whether Sisense was doing enough to protect sensitive data entrusted to it by customers, such as whether the massive volume of stolen customer data was ever encrypted while at rest in these Amazon cloud servers.
It is clear, however, that unknown attackers now have all of the credentials that Sisense customers used in their dashboards.
The breach also makes clear that Sisense is somewhat limited in the clean-up actions that it can take on behalf of customers, because access tokens are essentially text files on your computer that allow you to stay logged in for extended periods of time — sometimes indefinitely. And depending on which service we’re talking about, it may be possible for attackers to re-use those access tokens to authenticate as the victim without ever having to present valid credentials.
Beyond that, it is largely up to Sisense customers to decide if and when they change passwords to the various third-party services that they’ve previously entrusted to Sisense.
Earlier today, a public relations firm working with Sisense reached out to learn if KrebsOnSecurity planned to publish any further updates on their breach (KrebsOnSecurity posted a screenshot of the CISO’s customer email to both LinkedIn and Mastodon on Wednesday evening). The PR rep said Sisense wanted to make sure they had an opportunity to comment before the story ran.
But when confronted with the details shared by my sources, Sisense apparently changed its mind.
“After consulting with Sisense, they have told me that they don’t wish to respond,” the PR rep said in an emailed reply.
Update, 6:49 p.m., ET: Added clarification that Sisense is using a self-hosted version of Gitlab, not the cloud version managed by Gitlab.com.
Also, Sisense’s CISO Dash just sent an update to customers directly. The latest advice from the company is far more detailed, and involves resetting a potentially large number of access tokens across multiple technologies, including Microsoft Active Directory credentials, GIT credentials, web access tokens, and any single sign-on (SSO) secrets or tokens.
The full message from Dash to customers is below:
“Good Afternoon,
We are following up on our prior communication of April 10, 2024, regarding reports that certain Sisense company information may have been made available on a restricted access server. As noted, we are taking this matter seriously and our investigation remains ongoing.
Our customers must reset any keys, tokens, or other credentials in their environment used within the Sisense application.
Specifically, you should:
– Change Your Password: Change all Sisense-related passwords on http://my.sisense.com
– Non-SSO:
– Replace the Secret in the Base Configuration Security section with your GUID/UUID.
– Reset passwords for all users in the Sisense application.
– Logout all users by running GET /api/v1/authentication/logout_all under Admin user.
– Single Sign-On (SSO):
– If you use SSO JWT for the user’s authentication in Sisense, you will need to update sso.shared_secret in Sisense and then use the newly generated value on the side of the SSO handler.
– We strongly recommend rotating the x.509 certificate for your SSO SAML identity provider.
– If you utilize OpenID, it’s imperative to rotate the client secret as well.
– Following these adjustments, update the SSO settings in Sisense with the revised values.
– Logout all users by running GET /api/v1/authentication/logout_all under Admin user.
– Customer Database Credentials: Reset credentials in your database that were used in the Sisense application to ensure continuity of connection between the systems.
– Data Models: Change all usernames and passwords in the database connection string in the data models.
– User Params: If you are using the User Params feature, reset them.
– Active Directory/LDAP: Change the username and user password of users whose authorization is used for AD synchronization.
– HTTP Authentication for GIT: Rotate the credentials in every GIT project.
– B2D Customers: Use the following API PATCH api/v2/b2d-connection in the admin section to update the B2D connection.
– Infusion Apps: Rotate the associated keys.
– Web Access Token: Rotate all tokens.
– Custom Email Server: Rotate associated credentials.
– Custom Code: Reset any secrets that appear in custom code Notebooks.
If you need any assistance, please submit a customer support ticket at https://community.sisense.com/t5/support-portal/bd-p/SupportPortal and mark it as critical. We have a dedicated response team on standby to assist with your requests.
At Sisense, we give paramount importance to security and are committed to our customers’ success. Thank you for your partnership and commitment to our mutual security.
Regards,
Sangram Dash
Chief Information Security Officer”
As head of the Cisco Trust Office, Matt Fussa leads a global team that partners with government agencies, regulators, and customers to help shape cybersecurity regulation and manage cyber risk. He is… Read more on Cisco Blogs
The U.S. government agency in charge of improving the nation’s cybersecurity posture is ordering all federal agencies to take new measures to restrict access to Internet-exposed networking equipment. The directive comes amid a surge in attacks targeting previously unknown vulnerabilities in widely used security and networking appliances.
Under a new order from the Cybersecurity and Infrastructure Security Agency (CISA), federal agencies will have 14 days to respond to any reports from CISA about misconfigured or Internet-exposed networking equipment. The directive applies to any networking devices — such as firewalls, routers and load balancers — that allow remote authentication or administration.
The order requires federal departments to limit access so that only authorized users on an agency’s local or internal network can reach the management interfaces of these devices. CISA’s mandate follows a slew of recent incidents wherein attackers exploited zero-day flaws in popular networking products to conduct ransomware and cyber espionage attacks on victim organizations.
Earlier today, incident response firm Mandiant revealed that since at least October 2022, Chinese cyber spies have been exploiting a zero-day vulnerability in many email security gateway (ESG) appliances sold by California-based Barracuda Networks to hoover up email from organizations using these devices.
Barracuda was alerted to the exploitation of a zero-day in its products in mid-May, and two days later the company pushed a security update to address the flaw in all affected devices. But last week, Barracuda took the highly unusual step of offering to replace compromised ESGs, evidently in response to malware that altered the systems in such a fundamental way that they could no longer be secured remotely with software updates.
According to Mandiant, a previously unidentified Chinese hacking group was responsible for exploiting the Barracuda flaw, and appeared to be searching through victim organization email records for accounts “belonging to individuals working for a government with political or strategic interest to [China] while this victim government was participating in high-level, diplomatic meetings with other countries.”
When security experts began raising the alarm about a possible zero-day in Barracuda’s products, the Chinese hacking group altered their tactics, techniques and procedures (TTPs) in response to Barracuda’s efforts to contain and remediate the incident, Mandiant found.
Mandiant said the attackers will continue to change their tactics and malware, “especially as network defenders continue to take action against this adversary and their activity is further exposed by the infosec community.”
Meanwhile, this week we learned more details about the ongoing exploitation of a zero-day flaw in a broad range of virtual private networking (VPN) products made by Fortinet — devices many organizations rely on to facilitate remote network access for employees.
On June 11, Fortinet released a half-dozen security updates for its FortiOS firmware, including a weakness that researchers said allows an attacker to run malware on virtually any Fortinet SSL VPN appliance. The researchers found that just being able to reach the management interface for a vulnerable Fortinet SSL VPN appliance was enough to completely compromise the devices.
“This is reachable pre-authentication, on every SSL VPN appliance,” French vulnerability researcher Charles Fol tweeted. “Patch your #Fortigate.”
In details published on June 12, Fortinet confirmed that one of the vulnerabilities (CVE-2023-27997) is being actively exploited. The company said it discovered the weakness in an internal code audit that began in January 2023 — when it learned that Chinese hackers were exploiting a different zero-day flaw in its products.
Shodan.io, the search engine made for finding Internet of Things devices, reports that there are currently more than a half-million vulnerable Fortinet devices reachable via the public Internet.
The new cybersecurity directive from CISA orders agencies to remove any networking device management interfaces from the internet by making them only accessible from an internal enterprise network (CISA recommends an isolated management network). CISA also says agencies should “deploy capabilities, as part of a Zero Trust Architecture, that enforce access control to the interface through a policy enforcement point separate from the interface itself (preferred action).”
Security experts say CISA’s directive highlights the reality that cyberspies and ransomware gangs are making it increasingly risky for organizations to expose any devices to the public Internet, because these groups have strong incentives to probe such devices for previously unknown security vulnerabilities.
The most glaring example of this dynamic can be seen in the frequency with which ransomware groups have discovered and pounced on zero-day flaws in widely-used file transfer applications. One ransomware gang in particular — Cl0p — has repeatedly exploited zero day bugs in various file transfer appliances to extort tens of millions of dollars from hundreds of ransomware victims.
On February 2, KrebsOnSecurity broke the news that attackers were exploiting a zero-day vulnerability in the GoAnywhere file transfer appliance by Fortra. By the time security updates were available to fix the vulnerability, Cl0p had already used it to steal data from more than a hundred organizations running Fortra’s appliance.
According to CISA, on May 27, Cl0p began exploiting a previously unknown flaw in MOVEit Transfer, a popular Internet-facing file transfer application. MOVEit parent Progress Software has since released security updates to address the weakness, but Cl0p claims to have already used it to compromise hundreds of victim organizations. TechCrunch has been tracking the fallout from victim organizations, which range from banks and insurance providers to universities and healthcare entities.
The always on-point weekly security news podcast Risky Business has recently been urging organizations to jettison any and all FTP appliances, noting that Cl0p (or another crime gang) is likely to visit the same treatment on other FTP appliance vendors.
But that sound advice doesn’t exactly scale for mid-tier networking devices like Barracuda ESGs or Fortinet SSL VPNs, which are particularly prominent in small to mid-sized organizations.
“It’s not like FTP services, you can’t tell an enterprise [to] turn off the VPN [because] the productivity hit of disconnecting the VPN is terminal, it’s a non-starter,” Risky Business co-host Adam Boileau said on this week’s show. “So how to mitigate the impact of having to use a domain-joined network appliance at the edge of your network that is going to get zero-day in it? There’s no good answer.”
Risky Business founder Patrick Gray said the COVID-19 pandemic breathed new life into entire classes of networking appliances that rely on code which was never designed with today’s threat models in mind.
“In the years leading up to the pandemic, the push towards identity-aware proxies and zero trust everything and moving away from this type of equipment was gradual, but it was happening,” Gray said. “And then COVID-19 hit and everybody had to go work from home, and there really was one option to get going quickly — which was to deploy VPN concentrators with enterprise features.”
Gray said the security industry had been focused on building the next generation of remote access tools that are more security-hardened, but when the pandemic hit organizations scrambled to cobble together whatever they could.
“The only stuff available in the market was all this old crap that is not QA’d properly, and every time you shake them CVEs fall out,” Gray remarked, calling the pandemic, “a shot in the arm” to companies like Fortinet and Barracuda.
“They sold so many VPNs through the pandemic and this is the hangover,” Gray said. “COVID-19 extended the life of these companies and technologies, and that’s unfortunate.”
The legislation aims to bolster the Union’s cyber-resilience and enhance its capabilities to prepare for, detect and respond to incidents
The post The EU’s Cyber Solidarity Act: Security Operations Centers to the rescue! appeared first on WeLiveSecurity
Peter is an IT manager for a technology manufacturer that got hit with a Russian ransomware strain called “Zeppelin” in May 2020. He’d been on the job less than six months, and because of the way his predecessor architected things, the company’s data backups also were encrypted by Zeppelin. After two weeks of stalling their extortionists, Peter’s bosses were ready to capitulate and pay the ransom demand. Then came the unlikely call from an FBI agent. “Don’t pay,” the agent said. “We’ve found someone who can crack the encryption.”
Peter, who spoke candidly about the attack on condition of anonymity, said the FBI told him to contact a cybersecurity consulting firm in New Jersey called Unit 221B, and specifically its founder — Lance James. Zeppelin sprang onto the crimeware scene in December 2019, but it wasn’t long before James discovered multiple vulnerabilities in the malware’s encryption routines that allowed him to brute-force the decryption keys in a matter of hours, using nearly 100 cloud computer servers.
In an interview with KrebsOnSecurity, James said Unit 221B was wary of advertising its ability to crack Zeppelin ransomware keys because it didn’t want to tip its hand to Zeppelin’s creators, who were likely to modify their file encryption approach if they detected it was somehow being bypassed.
This is not an idle concern. There are multiple examples of ransomware groups doing just that after security researchers crowed about finding vulnerabilities in their ransomware code.
“The minute you announce you’ve got a decryptor for some ransomware, they change up the code,” James said.
But he said the Zeppelin group appears to have stopped spreading their ransomware code gradually over the past year, possibly because Unit 221B’s referrals from the FBI let them quietly help nearly two dozen victim organizations recover without paying their extortionists.
In a blog post published today to coincide with a Black Hat talk on their discoveries, James and co-author Joel Lathrop said they were motivated to crack Zeppelin after the ransomware gang started attacking nonprofit and charity organizations.
“What motivated us the most during the leadup to our action was the targeting of homeless shelters, nonprofits and charity organizations,” the two wrote. “These senseless acts of targeting those who are unable to respond are the motivation for this research, analysis, tools, and blog post. A general Unit 221B rule of thumb around our offices is: Don’t [REDACTED] with the homeless or sick! It will simply trigger our ADHD and we will get into that hyper-focus mode that is good if you’re a good guy, but not so great if you are an ***hole.”
The researchers said their break came when they understood that while Zeppelin used three different types of encryption keys to encrypt files, they could undo the whole scheme by factoring or computing just one of them: An ephemeral RSA-512 public key that is randomly generated on each machine it infects.
“If we can recover the RSA-512 Public Key from the registry, we can crack it and get the 256-bit AES Key that encrypts the files!” they wrote. “The challenge was that they delete the [public key] once the files are fully encrypted. Memory analysis gave us about a 5-minute window after files were encrypted to retrieve this public key.”
Unit 221B ultimately built a “Live CD” version of Linux that victims could run on infected systems to extract that RSA-512 key. From there, they would load the keys into a cluster of 800 CPUs donated by hosting giant Digital Ocean that would then start cracking them. The company also used that same donated infrastructure to help victims decrypt their data using the recovered keys.
A typical Zeppelin ransomware note.
Jon is another grateful Zeppelin ransomware victim who was aided by Unit 221B’s decryption efforts. Like Peter, Jon asked that his last name and that of his employer be omitted from the story, but he’s in charge of IT for a mid-sized managed service provider that got hit with Zeppelin in July 2020.
The attackers that savaged Jon’s company managed to phish credentials and a multi-factor authentication token for some tools the company used to support customers, and in short order they’d seized control over the servers and backups for a healthcare provider customer.
Jon said his company was reluctant to pay a ransom in part because it wasn’t clear from the hackers’ demands whether the ransom amount they demanded would provide a key to unlock all systems, and that it would do so safely.
“They want you to unlock your data with their software, but you can’t trust that,” Jon said. “You want to use your own software or someone else who’s trusted to do it.”
In August 2022, the FBI and the Cybersecurity & Infrastructure Security Agency (CISA) issued a joint warning on Zeppelin, saying the FBI had “observed instances where Zeppelin actors executed their malware multiple times within a victim’s network, resulting in the creation of different IDs or file extensions, for each instance of an attack; this results in the victim needing several unique decryption keys.”
The advisory says Zeppelin has attacked “a range of businesses and critical infrastructure organizations, including defense contractors, educational institutions, manufacturers, technology companies, and especially organizations in the healthcare and medical industries. Zeppelin actors have been known to request ransom payments in Bitcoin, with initial amounts ranging from several thousand dollars to over a million dollars.”
The FBI and CISA say the Zeppelin actors gain access to victim networks by exploiting weak Remote Desktop Protocol (RDP) credentials, exploiting SonicWall firewall vulnerabilities, and phishing campaigns. Prior to deploying Zeppelin ransomware, actors spend one to two weeks mapping or enumerating the victim network to identify data enclaves, including cloud storage and network backups, the alert notes.
Jon said he felt so lucky after connecting with James and hearing about their decryption work, that he toyed with the idea of buying a lottery ticket that day.
“This just doesn’t usually happen,” Jon said. “It’s 100 percent like winning the lottery.”
By the time Jon’s company got around to decrypting their data, they were forced by regulators to prove that no patient data had been exfiltrated from their systems. All told, it took his employer two months to fully recover from the attack.
“I definitely feel like I was ill-prepared for this attack,” Jon said. “One of the things I’ve learned from this is the importance of forming your core team and having those people who know what their roles and responsibilities are ahead of time. Also, trying to vet new vendors you’ve never met before and build trust relationships with them is very difficult to do when you have customers down hard now and they’re waiting on you to help them get back up.”
A more technical writeup on Unit 221B’s discoveries (cheekily titled “0XDEAD ZEPPELIN”) is available here.
Editor’s note: While this topic isn’t entirely security-specific, Trend Micro leader William Malik, has career expertise on the trending topic and shared his perspective.
——
There was a provocative report recently that the Governor of New Jersey told reporters that the state of New Jersey needed COBOL programmers. The reason was that the number of unemployment claims had spiked, and the legacy system running unemployment claims had failed. That 40-year-old system was written in COBOL, so the conclusion was that the old language had finally given out. Hiring COBOL programmers would let the State update and modernize the application to handle the increase in load.
This might be the problem, but it probably is not. Here’s why.
From these points, we learn the following lessons.
Software Doesn’t Wear Out
Logic is indelible. A computer program is deterministic. It will do exactly what you tell it to do, even if what you tell it to do isn’t precisely what you meant it to do. Code never misbehaves – but your instructions may be incorrect. That’s why debugging is such a hard problem.
Incidentally, that’s also why good developers usually make lousy testers. The developer focuses her mind on one thing – getting a bunch of silicon to behave. The tester looks for faults, examines edge conditions, limit conditions, and odd configurations of inputs and infrastructure to see how things break. The two mindsets are antithetical.
Once a piece of software has been in production long enough, the mainline paths are usually defect free. In fact, the rest of the code may be a hot mess, but that stuff doesn’t get executed so those defects are latent and do not impact normal processing. Ed Adams published a report in 1984 titled “Optimizing Preventative Service for Software Products” (https://ieeexplore.ieee.org/document/5390362, originally published in the IBM Journal of Research and Development, v 28, n 1). He concluded that once a product has been in production for a sufficient time, it was safer to leave it alone. Installing preventative maintenance was likely to disrupt the system. Most IT organizations know this, having learned the hard way. “If it ain’t broke, don’t fix it” is the mantra for this wisdom.
As a corollary, new software has a certain defect rate. Fixes to that software typically have a defect rate ten times greater. So if a typical fix is large enough, you put in a new bug for every bug you take out.
Computers Are Constrained
All computers have constraints. The relative amount of resources mean some computers are better for some workloads than others. For mainframes, the typical constraint is processing power. That’s why mainframes are tuned to run at 100% utilization, or higher. (How do you get past 100% utilization? Technically, of course, you can’t. But what the measurements are showing you is how much work is ready to run, waiting for available processing power. The scale actually can go to 127%, if there’s enough work ready.)
Different types of computers have different constraints. Mainframes run near 100% utilization – the CPU is the most expensive and constrained resource. PCs on the other hand never get busy. No human can type fast enough to drive utilization above a few percent. The constrained resource on PCs is typically disk storage. That’s why different types of computers do better at different types of work. PCs are great for user interface stuff. Mainframes are perfect for chewing through a million database records. By chance we developed mainframes first; that’s not an indictment of either type, Both are useful.
Computers Can Run Out of Resources
Any IT infrastructure has a design point for load. That is, when you put together a computer you structure it to meet the likely level of demand on the system. If you over-provision it, you waste resources that will never be used. If you under-provision it, you will not meet your service level agreements. So when you begin, you must know what the customers – your users – expect in terms of response time, number of concurrent transactions, database size, growth rates, network transaction load, transaction mix, computational complexity of transaction types, and so on. If you don’t specify what your targets are for these parameters, you probably won’t get the sizing right. You will likely buy too much of one resource or not enough of another.
Note that cloud computing can help – it allows you to dynamically add additional capacity to handle peak load. However, cloud isn’t a panacea. Some workloads don’t flex that much, so you spend extra money for flexibility for a capability that you can provide more economically and efficiently if it were in-house.
Add Capacity in Balance
When I was in high school our physics teacher explained that temperature wasn’t the same as heat. He said “Heat is the result of a physical or chemical reaction. Temperature is simply the change in heat over the mass involved.” One of the kids asked (snarkily) “Then why don’t drag racers have bicycle tires on the back?” The teacher was caught off guard. The answer is that the amount of heat put into the tire is the same regardless of its size, but the temperature was related to the size of the area where the tire touched the road. A bicycle tire has only about two square inches on the pavement, a fat drag tire has 100 square inches or more. So putting the same amount of horsepower spinning the tire will cause the bicycle tire’s temperature to rise about 50 times more than the gumball’s will.
When you add capacity to a computing system, you need to balance related capacity elements or you’ll be wasting money. Doubling the processor’s power (MHz or MIPS) without proportionately increasing the memory or network capacity simply moves the constraint from one place to another. What used to be a system with a flat-out busy CPU now becomes a system that’s waiting for work with a queue at the memory, the disk drive, or the network card.
Adding Staff Makes Things Worse
Increasing any resource creates potential problems of its own, especially of the system’s underlying architecture is ignored. Fore the software development process (regardless of form) one such resource is staff. The book “The Mythical Man-Month” by Fred Brooks (https://www.barnesandnoble.com/w/the-mythical-man-month-frederick-p-brooks-jr/1126893908) discusses how things go wrong.
The core problem is adding more people require strong communications and clear goals. Too many IT projects lack both. I once was part of an organization that consulted on a complex application rewrite – forty consultants, hundreds of developers, and very little guidance. The situation degenerated rapidly when the interim project manager decided we shouldn’t waste time on documentation. A problem would surface, the PM would kick off as task force, hold a meeting, and send everybody on their way. After the meeting, people would ask what specific decisions had been reached, but since there were no minutes, nobody could be sure. That would cause the PM to schedule another meeting, and so on. Two lessons I learned concerns meetings:
When you add staff, you must account for the extra overhead managing the activities of each person, and establish processes to monitor changes that every participant must follow. Scrum is an excellent way of flattening potentially harmful changes. By talking face to face regularly, the team knows everything that’s going on. Omit those meetings or rely on second-hand reports and the project is already off the rails. All that remains is to see how far things go wrong before someone notices.
In Conclusion …
If you have a computer system that suddenly gets a huge spike in load, do these things first:
If you are unable to resolve the capacity constraints with these steps, examine the programs for internal limitations:
Note that if you do not have (or cannot find) the relevant documentation, you will need to examine the source code. At this point, you may need to bring in a small set of experts in the programming language to recreate the relevant documentation. Handy hint: before you start working on the source code, regenerate the load modules and compare them with the production stuff to identify any patches or variance between what’s in the library and what’s actually in production.
Bringing in a bunch of people before going through this analysis will cause confusion and waste resources. While to an uninformed public it may appear that something is being done, the likelihood is that what is actually being done will have to be expensively undone before the actual core problem can be resolved. Tread lightly. Plan ahead. State your assumptions, then verify them. Have a good plan and you’ll work it out. Remember, it’s just ones and zeros.
What do you think? Let me know in the comments below, or @WilliamMalikTM.
The post “We Need COBOL Programmers!” No, You Probably Don’t appeared first on .
FreshRSS 