In this, third in a series of Crucial Questions posts I’m going to focus on the questions from CISOs and security teams. This builds on many related topics covered in the two prior posts on crucial questions from CIOs/CTOs and Boards and executives.
Remember, these are not the questions I necessarily think CISOs should be asking, but they are what I hear a lot. These are not in any ranked order and I’ve tried not to bias the questions as to what I would ask. The nature of what my role is and the context of the interactions I have with CISOs may well drive the dialog to be more cloud, technology modernization and security products related - but I also pick up questions from various CISO events I attend. One final disclaimer, this post is in no way meant to be exhaustive, it just touches on the wave-tops of these topics.
[Note: in this post I will simply say CISOs to mean CISO as well as other types of security leaders and their teams.]
1. Hybrid Multi-Cloud
How do we manage security across our hybrid on-premise and multi-cloud / multi-SaaS environments?
This, as you would expect, is an increasingly common question and concern from CISOs. Most organizations, large or small, are becoming a combination of service providers and on-premise technology inter-woven in various ways. One of the bigger challenges for CISOs is not just to ensure that each individual service is appropriately secured but that the collection of those services that make up a business or mission process is secure. It’s an even bigger challenge to assure the mitigation of other risks across resilience, compliance, privacy, data governance, and other domains.
Specifically on cloud, there are now fewer organizations that rely on one cloud service provider (CSP). Also, while there are a growing number of cloud-centric companies, the majority of organizations still have significant on-premise technology. There are a number of drivers for organizations taking a multi-cloud approach. The most common I see are:
A business unit or the overall organization needs a capability where a specific CSP has a comparative advantage.
A new CSP is inherited as a provider to an acquired company.
A CSP is introduced to provide competition and price pressure on the incumbent(s).
An additional CSP provides resilience in some real or perceived way.
For performance needs, where the current CSP can’t scale to meet the organization’s growing operational needs.
Where an incumbent has not resolved security or compliance issues effectively enough.
In thinking about how to manage the reality of hybrid multi-cloud risk, there’s no way I can give a thorough answer to this in a short blog post but just to give a flavor of what I say:
Establish central standards for minimum control expectations on each cloud provider and implement that as part of a set of secure "landing zones" or blueprints which provide the default controlled rails for business units to operate on.
Encode those control expectations as default product configurations that are enforced declaratively (controls-as-code) and are constantly monitored for adherence (continuous controls monitoring). Pay particular attention to potentially insecure defaults in certain products from some CSPs. Utilize the CSPs recommendations and reference architectures, they all have pretty good documentation, in some cases backed by configuration code.
Most CSPs' Identity and Access Management (IAM) constructs are quite complex because they need to cover a lot of implementation scope. Many CSPs have been introducing ABAC-like concepts to join RBAC approaches and other administrative conveniences like tags that can be used to overlay different administrative groupings. These simplify how you can enforce policies in single CSPs but can also be used to overlay some consistency on how you do this across all your CSPs. In multi-cloud / multi-SaaS environments it’s useful to minimize the number of identity providers you use through federation / BYOID approaches. You can extract (with your own capability or a third party tool) CSP and SaaS identity, access and role management information to continuously assess whether that is still meeting your policy intent. But on federation, be careful of fault or incident blast radius extent in case your one identity provider becomes a single point of compromise. Explore ways to limit some of the scope of that federation or require separate step-up authentication for critical activities.
Establish approaches across each CSP to assure defense in depth from configuration errors as well as attacks. Most CSPs can provide separation of duties for different teams e.g. developers, cloud engineering, network engineering, and security over different resources types from policy configuration, VPC administration, security settings and so on. Grouping specific critical privileges from each CSP into an administrative zone under the control of security is useful to segregate duties among teams and people.
Collect logs and other sensory collection from each CSP and SaaS environment into a central scalable repository and systematically hunt for anomalous activities and triggers of well defined events. Each CSP has their own versions of security event and configuration management and it is worth extracting key alerts and events from those into a common alerting framework using one of the CSPs products, a third party product or your existing monitoring framework. Be careful on scaling though, one of the great things about cloud is the breadth and depth of sensory coverage which heightens security observability but can easily overwhelm traditional log / event data management systems. I would tend to worry less about the illusory “single pane of glass” but rather focus on ensuring a common operating picture. Most of the time I’ve seen companies focus on the “single pane of glass” all they get is a “single pain of glass” that needs to be pushed to one side to get to the functions in the tools it abstracted.
Develop a comprehensive training program that decouples cloud security principles and generic control objectives from CSP specific control functions. For those CSP specific control functions utilize the often great CSP provided training, labs and simulations not just for configuration but for preparing responses to various scenarios.
Don’t try too hard to fully align your existing on-premise environment with cloud especially if that will be deprecated over time - but be very careful on the interface points. Have well defined ingress/egress to and from cloud. But, of course, start quickly adopting cloud-like security approaches as much as you can on-premise. This includes least privilege, segmentation, RBAC and ABAC access management patterns, improved software delivery and integration and VM / container security such that only known good software can execute in the run time environment.
There’s often discussion of how CISOs need to change their security mindset to adopt a cloud way of thinking about security. I actually think this is the wrong way to think about it. Rather, you should go back to first principles of security and realize that you can now implement those principles much more effectively in the cloud. In other words, we’ve spent decades constructing work-arounds to the on-premise inability to implement many of those principles. Now we can implement them in the cloud - so it’s less about learning a new way of working and more of unlearning the compromises we had to make.
Finally, hold your SaaS vendors to the same standards as your CSPs. In a lot of organizations the SaaS vendors get a lightweight security review and the CSPs get an, appropriate, architectural colonoscopy. But many of the SaaS vendors have some of your most critical data and increasingly their FaaS (Function as a Service) style of operation can be a center for workloads as well. It is worth doing deeper dives into their key controls especially around software security, encryption coverage and internal privileged access controls of their support and engineering teams.
2. Threat Intelligence
How can we obtain, curate and act on threat intelligence in more effective ways?
A lot of CISOs have shifted their world view from being focused on threat intelligence as simply a collection of feeds which supply Indicators of Compromise (IOCs) or other signals into their security operations function. In my experience leading CISOs are much more focused on strategic threat intelligence - where they are interested in learning about the evolution of threat actor TTPs (Tactics, Techniques, and Procedures). They are deeply interested in specific incidents that have occurred in other organizations that would signal changes in TTPs or provide other lessons learnt that would have them prioritize changes in their defensive posture to stay ahead of those threats.
So in answering this I remind people to focus on the strategic goals of intelligence:
First, let’s remember there are, in simple terms, two broad categories of threat intelligence:
Macro threat intelligence. Information on attacker goals, capabilities & evolving TTPs. Use this to adjust defenses to make life more difficult for the adversary and shape their economics (attackers have bosses and budgets too). Aim to eliminate whole classes of attacks. For macro you need to feed it into your risk decision making process as fast as possible and increase the speed of adjusting defenses.
Micro threat intelligence. Information about specific attacks, signatures, indicators of compromise and other selectors / data. Aim to eliminate or detect / respond to specific attacks. Information about threats, itself, is necessary but not sufficient. In both cases you need to be capable of doing something with it. For micro threat intelligence you need to feed this into your defensive operations as fast as possible - in as fully an automated way as you can. Work to improve the ingest speed and coverage of this into your preventive controls and your detective sensor grid.
Threat intelligence is often undervalued due to organizations' inability to process it (perhaps fueled by over marketing of what it can do - by vendors or pundits). If you buy something or consume some capability you have to be equipped to use it. There’s no point buying some feed if you can't do anything with it. I’ve seen a lot of organizations bemoan some Government entity or some vendor not providing volumes of intelligence but then not introspect to wonder if such intelligence was provided would they even have the ability to consume, analyze, and act on it.
Finally, don’t confine intelligence to be just about threats. It is worth developing a scenario planning approach so that in your use of wider intelligence you are joining that with some horizon scanning for risks and other changes in the world that could foretell other needed adaptation.
3. Security Monitoring
How can I scale security monitoring to deal with increased attack surface and increased sensory coverage?
This has become a more pervasive question from CISOs largely I think in line with the increased digitization of people’s businesses and the complexity of their platforms. This extends, digitally, upstream from customers and downstream into a complex web of suppliers - all pieced together with APIs. This means that there’s not only more to monitor but there’s also more possibilities in terms of sensor placement. But, above all, there’s a massive amount of other input in the form of what you could call the “digital exhaust” kicked off from all the interactions of these various products and services. This can range from basic logs to more sophisticated aggregated events from IT and business service observability tools. In many respects we’re in the golden age of security observability because of this. I haven’t done a full study, but my informed intuition is that in most organizations of all the potentially useful information for security monitoring 95% is that exhaust emitted by the environment and 5% is that specifically collected from designated security sensors. The question is what are the right places to dig in this 95% and do you even have the capacity to deal with that and make sense of it. So in answering this I inevitably focus on the following:
First, remember your best security analysis tools might not be what you can buy from security vendors. Some of the best tools for security monitoring and analytics (or fraud and other forms of wide-scale risk monitoring) can be the large scale data analytics platforms from various CSPs or other providers that run on CSPs. These are useful to join with security monitoring products that are pre-configured with specific analytics and fueled by provider-sourced threat intelligence. But if you find yourself in a situation where you need multiple security monitoring products then you might be better off with one specialist product and then amalgamate all your other needs to a general purpose data analytics platform.
Similarly, when you look at the use of AI (generally speaking ML based AI) there are many security vendors that have a particular trained model for a specific detection objective which is encoded in a product. The problem with many of these products, though, is that they are not useful to detect other situations and patterns. In many cases it’s not unreasonable to use a general purpose AI/ML system from a CSP or other vendor. These have a base environment you can adapt for multiple purposes with increasingly useful tooling even for organizations that don’t have high-end ML specialists on staff. If you are not careful you can find yourself with dozens of ML based products that only do a specific detection, or worse are static models that aren’t updated based on the learning opportunities in your own environment with your own factors.
Generic large scale data analytic solutions more effectively support data governance, workflow and lineage challenges that are vital when you need to test, attest or certify the effectiveness of the data pipeline that supports your monitoring environment. I’ve seen many examples (not just in security) where small-scale niche vendor products fail to operate effectively within an environment where data controls and model validation are important - and that’s more environments each day. If you're in a regulated critical infrastructure sector don't be surprised to be questioned by your regulator(s) on how you trained your AI, controlled the lineage of the training data, assured the end to end data pipeline for accuracy and completeness and have a reasoned approached for model validation and explainability. Very few (and I'm being generous here) niche security AI/ML products can support this wider governance need whereas, at least some, CSPs AI/ML products have this apparatus built-in.
The huge volumes of data to consume and the potential events to triage / handle means you have to automate as much as possible. But, for many environments it’s not enough to simply automate. In many cases you also want to reduce the need for that automation by constructing autonomic approaches where your systems are capable of self-defense and intrinsic response. There’s a great document that describes this approach here.
As mentioned above, we all now have a vast collection of sensory data available to us from the wider business, mission and IT environments. If you go back a decade or more most of security monitoring was data collected from the placement of specialized security sensors (hardware or software) and from logs emanating from specific policy enforcement points (e.g. firewalls, proxies, endpoint agents, O/S security subsystem). Now we have a rich sensory input that can be consumed from all and every layer of the stack linked with other observability needs. The challenge now is to deal with that volume and make sense of it. It’s also important to take opportunities to optimize our environments for detection, for example, with service mesh and other segmentation approaches. We can, at higher levels in the stack, have a more intrinsic understanding of what is anomalous so we can optimize data collection and tune the balance of false positive / false negatives better because of system design.
You should also couple control deployment with detection. Continuous control monitoring can assure that the controls you expect to have are still operating effectively in the right places. But, you also have an opportunity to build detection rules around those controls. For example, if you have a control that blocks a certain type of traffic crossing from one domain / zone to another then you should also have a sensor or some observability that is primed to detect if that is ever not the case. That should measure zero constantly, by definition. If it ever doesn’t then something has gone wrong with the control - and if your continuous control monitoring hasn’t also picked that up then something is terribly wrong. This can be the ultimate expression of balancing prevention and detection.
Now, this brings up an interesting point about detections that read zero in their steady state. If something reads zero as normal then how do you know it is working? In some nuclear power situations where the environment is radiation hardened, to the extent background radiation can’t get in, then the Geiger-counters read zero, so you have to put small/harmless radiation sources in them so they register a little so workers can see the sensors are working. In a situation closer to home, years ago I uncovered an issue where an array of network intrusion detection sensors (IDS) failed to spot an event. It turned out the sensors hadn’t been detecting anything for a month because there was a bad network change that took the taps off-line that were coupled to the sensors. I know, this is pretty incredulous given how noisy network IDS used to be but the team had done a pretty good job of tuning so zero readings for some periods were not unusual. But again, zero is bad because you don’t know if zero is good (no events) or zero is bad (broken sensor). After this we created a series of tools to generate synthetic events (akin to system heartbeats) so that all sensors would continuously read something and were complemented by alarms that would go off if any sensor did read zero. This was well before the “attack simulation” range of products that now exist that can achieve a similar result. Incidentally, many organizations apply the same logic to test business controls by injecting synthetic faults / frauds and other events to constantly validate their detection pipelines are working and to build the muscle-memory of responders.
Finally, it surprises me how few organizations look at incidents or close calls that have happened at other organizations and rigorously examine “what if this happened to us” - would we have seen it, would we have prioritized response, would we have preempted impact?
4. Workforce Challenges
How do I fill the positions I have with the right skills and people? How do I nurture my leaders and build a succession plan?
This is another dominant question because of issues many organizations face in hiring. However, most organizations are realizing that it’s not simply a numbers game of: do we have enough cybersecurity workers to fill the positions we have? No, most organizations now realize this is a more balanced problem of thinking whether the organization is structured the right way, is the team self-sustaining and developing, and are functions aligned to other parts of the organization. Essentially, is the CISO function operating with capability that is greater than the sum of its parts. In answering this question I inevitably can’t steer clear of talking about workforce scaling and balance:
Most of the dialog about the skills shortage in cybersecurity focuses, perhaps inevitably, on the all too simple answer of "let's create more cybersecurity professionals". This is usually coupled with (often unsubstantiated) claims that there are millions of open cybersecurity positions. The answer, thus, becomes fixated on how we create all those cybersecurity people - whatever a "cybersecurity person" is.
We need to equivalently focus on cyber-workforce productivity. If we need 10x more cybersecurity people to fill all those roles, perhaps if we could 10x the productivity of the people we already have then that should significantly address the issue. Productivity isn't just about automation / orchestration - it can also be stopping doing things, aligning control mitigation practices across different IT risks, auto-configuring, embedding testing, auto-generating tests from threat models, and ensuring the right people do the right jobs matched to the right skills.
We have to do a better job on adjusting our roles and organizations to create actual entry level positions. We can require our teams to create roles that are apprentice-like or use the breadth of our federated organizations to create entry-level security programs in support teams where the barriers to entry are less and opportunities for training are increased. Then within 12-months rotate them into the security team at entry level +1 if that’s what it takes.
Diversity, equity and inclusion (DEI) is vital in any risk program. Group-think is the mind killer for risk management and while you can create many process antidotes to group-think the best innate way is to have a team which brings a natural diversity of thought and experience to every situation. It is important to not just have diversity but to also foster an environment of inclusion where all those voices are actually heard.
In building a balanced team, follow the rule of thirds to ensure you have a range of operational, risk and technically specialist roles. Lack of such balance means you won’t be able to capitalize on the strengths you have on the team. For example, if you have immense technical depth but no business awareness to translate technical risks into how they affect business or mission goals then you’re unlikely to receive the support or funds to mitigate those risks. Similarly, if you have great business acumen but little technical depth in the team you’re only going to surface and resolve high level control issues not the deep-technical flaws that might exist in your environment. If you have technical and business depth but no operational capabilities then you might never scale or sustain your work.
In terms of leadership development, and succession planning overall, I’ve found this has become harder over time in many organizations. There’s a paradox here where organizations get better funded and larger but then create specialist stove-pipes from which leaders emerge who are well versed in those domains but have little other experience. If you’re not careful you can find yourself as a CISO with an absolutely amazing leadership team in their own right and own domains but for which those leaders might not be able to slot into other leadership positions and not be a viable successor for you. Some of the most successful CISOs are an accident of history having grown up in tandem with the maturing of the field so they were also able to learn as they went along. That’s not the case now so you have to explicitly plan for this through a range of techniques:
Occasionally rotate leaders between functions.
Assign stretch opportunities in an adjacent area while that leader remains in their role.
If your scale allows, have leaders of central teams drop into Business Unit CISO (so called BISO) or similar roles so they can build experience of the broader role with more support and at less challenging scale.
Involve your leaders in more of your day to day work or, ideally, delegate key parts of your role to multiple of them. This could range from large programs, external engagement through to parts of working with the Board.
Finally, let’s talk about remote work. I know this is a topic that attracts a lot of disagreement but what I’m finding is that balance is needed. Yes, we need to support more remote work to enable workforce flexibility and to hire talent in more places. But, it’s a mistake to think you can operate solely remotely without some structured approach to occasionally convening teams in person. Restructuring work around periods of remote or otherwise dispersed work coupled with more frequent and structured workshops, team events, collaboration days and such is important to create trust, sharing and deeper engagement than can be achieved in a purely on-line way. Since many organizations have created a hybrid mode of working I’ve seen an increased level of energy, peer and inter-team trust and a whole level of engagement that is hard to sustain in a 100% permanent remote mode. It’s up to your organization, culture and team dynamic as to what hybrid means. For some organizations it might be 2 days remote and 3 days in per week all the way to permanently remote with periodic on-premise team meetings.
5. Board and Executive Relationships
How do we keep the Board and executive leadership informed and on-side with our efforts?
As Boards and executives see the relentless parade of incidents in many organizations and are pressed by shareholders, other stakeholders, customers and regulators they, of course, place higher demands on CISOs and their teams to ensure cybersecurity and all the other technology, business information and enterprise risks are being managed effectively. This is another area where it is important to be balanced. Most organizations don’t have cybersecurity as their primary mission - they have business goals or operational missions to achieve. Leading CISOs don’t just focus on how they play defense - that’s now table stakes. Rather, they focus on how they support commercial and mission success. So when you think about the Board and executive relationships I tend to go to this space more than before:
Focus on Risk. Make sure you can answer this combination question: “What are the most significant risks to our most critical assets and business services, what controls mitigate those risks and who is continuously assessing whether those controls are in place and effective? What residual risks remain and who at what level of the organization deemed those acceptable with what compensating factors or risk transference? What executive management group regularly monitors the measured outcome of this process?” Notice that these questions don't contain the word cybersecurity. This is how you manage many types of risk and is something most Boards and Board Directors will be familiar with. It is, of course, deceptively simple to state but fiendishly difficult to answer well. It requires a significant amount of work to develop risk taxonomies, asset and service inventories, risk and continuous control monitoring and an evolving apparatus of risk governance.
Think Beyond Cyber. Cybersecurity is just one of many technology and information risks and shouldn’t be discussed in isolation, especially given most of the best mitigations for cybersecurity risk are also powerful mitigations for other technology risks, such as software and service lifecycle management, identity/access management, data governance, and highly resilient and monitored production services.
Business Initiatives - Embed Cyber in other's Content. There are many other discussions at Boards ranging from business initiatives, other risk and control reviews, strategic discussions, financial reviews, attestations, and so on. Work with your peer executives across business lines and control functions to make sure that relevant content on cybersecurity / technology risk appears in their Board content. Work to educate those leaders and prepare them for questions that come from that based on your experience of talking to the Board. This creates what you really desire, the shared accountability to mitigate these risks across the enterprise. It can be transformational if the Board is asking everyone they encounter about how cybersecurity is managed in the context of that activity / business process as opposed to just asking the CISO.
Measure Capabilities. Develop the means to show the Board what capabilities the organization has to manage cybersecurity / technology risk - not just the current status of the risks being managed. Our space is highly dynamic, so the major measure of an organization’s ability to manage risk is how well they are positioned to see what is coming and to respond to it well. I go into a lot more detail on this topic in this post.
Demonstrate Business Value. Not just in losses avoided, evidenced (hopefully) from showing incidents at other organizations could not happen to yours - but also showing adjacent commercial benefits such as improved customer experience, reduced insurance premiums, lower loss covering capital retained, increased IT productivity, enablement of new digital services and so on.
Present Jointly and Collaboratively. Unless you are in an independent audit or independent risk management capacity where you might be obliged to present independently, then strive to present jointly with your IT or other colleagues such as the CTO, CIO, Chief Digital Officer, Chief Data Officer or others. Presenting a unified front not only solidifies relationships in the preparation but also increases subsequent collaboration. Presenting jointly doesn’t necessarily mean presenting a singular opinion. It is totally acceptable (perhaps desirable) to illustrate differences in perspective between functions, it’s likely the CISO function might be more conservative. These should never be framed as personal disagreements but rather it is a good opportunity for these to presented as a difference in risk appetite or risk tolerance for which the Board can provide some useful input according to some well developed options.
Deliver Board Education. Make sure your Board is educated to a reasonable extent, not just for the designated members of the Board who have a background and expertise relevant to this risk. This doesn’t have to happen at the Board meeting itself. Some of the best education is one on one, at Board lunches or dinners and especially at the orientation of new Board members.
6. IT Modernization
How do I ensure IT prioritizes necessary security upgrades and IT systems modernization?
CISOs, and wider organizations, have mostly internalized either through wisdom or experience the need to invest to modernize IT onto architectures where security is built in, not bolted on. Many organizations have seen the poor results of investing a lot in cybersecurity products and deploying them on the foundation of sand that is legacy technology. So now the question is how should the CISO partner with IT to ensure that in the technology modernization efforts, on-premise, in the cloud or both you can ensure the right security controls are built in. Here are some considerations:
Establish a formal program to find and plot “architectural intercepts”. In the initial phase this is about making sure the CISO’s team are engaged in reviewing the security and architectural effectiveness of IT projects to make sure they are done well (mitigating risk) but, and this is the point, that they also seek opportunities to do things better. For example, IT might be implementing a new VM or container runtime environment for large parts of production. You will naturally want to make sure this is securely configured and managed but you’ll also want to seize upon the opportunity to make sure that such a runtime only executes known, approved and authentic (signed) software artifacts from your build systems. There may be other longer term architectural intercepts associated with new business projects or other initiatives. For example, it might be that a business unit is reengineering their entire means of digitally interacting with customers. Again, you’ll want to make sure this is secure but at the same time this reimagining of that interaction might present opportunities for improved controls around customer authentication - perhaps not just for risk reduction but for improved customer experience. This aspect of the work might require working with a different part of the project team than your team is typically used to such as having people work with the product team and UX designers as well as the IT team building what is designed.
Give up some control and embed that into other teams. If your goal is secure products not just security products then teams building products need self-contained security engineering expertise - not just consulting and testing from the CISO team. It might be your best way to improve and scale security is for there to be more security capabilities outside of your team embedded in other IT teams whether that is identity and access management, developer tools, default application architecture, or cloud infrastructure engineering.
Identify your core critical or foundational infrastructure components and make sure these have well defined and well funded roadmaps for future investments and upgrades and jointly sponsor such work between the CISO, CIO or CTO depending on the structure of your organization.
Mercilessly eliminate vendors who fail to get with your program who are a drag on all your other objectives. I remember one situation many years ago where we were eliminating all highly privileged desktop accounts, there was one application from a very large IT vendor that would have meant keeping Local Admin (Windows) privilege for 13,000 end users because of the deeply flawed architecture of the client side libraries of this particular database product (this was in the 2-tier client server days). This particular vendor paid lip-service to our needs and assumed our dependency on its incumbency meant that we’d do nothing and just accept the risk. Well, kudos to my colleagues the CIO and CTO we eliminated not just that product in less than 12 months to achieve our goal of eliminating Local Admin privileges but we then progressively removed that vendor from most of the rest of the environment over the coming years as well.
While the broader goal is to align on modernization efforts it is also important to make sure your environments are kept up to date, patched of course, but also within some reasonable window of the current product version so you don’t end up having your good legacy systems become bad stagnant systems. So, create a preventative maintenance budget of some percentage of each business unit's IT budget to invest in keeping things up to date. This is a useful source of funds to allocate for other types of necessary control updates driven by security, audit or external regulatory needs. The percentage itself can vary year on year according to risk-adjusted criteria.
7. Benchmarking
How do I know if I’m doing enough and if my risk profile is about right?
This comes up still, but interestingly a lot less than it used to. I think because many CISOs have realized it’s important to learn approaches from other organizations but your particular objectives are keyed off your own risk profile as opposed to what others do. So I try to steer clear of benchmarks unless they are directly tied to something you can learn from. I especially dislike budget benchmarks. Let me explain:
Be wary of budget benchmarking. Three simple reasons:
A budget is an input not an outcome. Security risk management needs to be centered on outcomes.
There is no agreed upon taxonomy of comparison. You’re never comparing apples to apples.
Incentives are misaligned. Budget comparisons aim to set minimum standards, "spend at least X”, but often good security programs become more efficient and spend less per unit of control (overall budget might increase as the organization grows but this should be sub-linear to the overall enterprise spend).
Our goal is to mitigate security risk in the best way for our organizations while balancing the needs of customers and the commercial purpose of the enterprise. Your risk is not my risk. Your business is not my business. Your threat outlook is not mine. Just because you and I spend roughly the same doesn’t mean we will get the same result, I might have different people, different issues, different established infrastructure and so on.
As an aside, if your leadership says things like "we will spend whatever it takes to make sure we have no security issues”, then when an event inevitably happens does that mean you didn’t actually spend enough despite what you said? I’ve also seen some organizations actually spend too much because of excess focus on input not outcomes which resulted in wasted product spend, conflicting projects, overlapping teams and unnecessary employee churn.
If the goal is to compare what are the most effective practices to drive improved security outcomes then some of the following are great places to start:
How much is security embedded in product design and development (shift left).
The extent to which controls are seamlessly embedded into business processes and supporting systems (ambient controls).
The degree of automation of repeatable activities (never send a human to do what a computer can do).
How frequently the organization is adapting its processes or architecture to reduce inherent risk (risk avoidance).
Whether the organizing is constantly taking new updates and features (the rising tide of cloud and service centric security).
Interestingly, when these are done well then they usually show reductions in expenditure per unit cost of control. If you're focused on budget benchmarks for which success equates to more spend you might, paradoxically, be disappointed with actual better outcomes.
Bottom line: I am regularly impressed with the cadre of CISOs leading security and in many cases broader risk management at most organizations. Their questions show a great desire to not just defend but also to drive their organizations’ commercial and mission goals forward. Many are doing this as true partners with their business and IT colleagues and have a, so called, seat at the table not just because they kicked the door in and sat down but because that seat was placed by their Boards and executives who realize how crucial the CISO and their team is to sustained success.