FME prepared a survey to learn about the community’s views on the topic of Conference Rankings and the FM Symposium. The survey ran until January 29 - February 28, 2026. We now report on the result of the survey. The report is also available as a pdf.
This report summarizes the results of a survey on conference rankings conducted among members of the formal methods research community and initiated by Formal Methods Europe (FME). The survey collected their about conference rankings, and the ICORE rankings in particular.
For context: recently, ICORE (also known as CORE) downgraded the FM symposium from rank A to rank B, despite the FME board having submitted an extensive documentation about FM’s standing and attractiveness for researchers in formal methods.
The survey is in three parts reflected in this report’s structure:
- A few questions about the respondents’ background and demographics
- Several multiple-choice questions about rankings and the FM symposium
- An optional open question for further comments
The survey collected 80 responses in anonymous form.1
Our (the FME board’s) interpretation of the survey results and discussion of implications for our strategy are interspersed throughout the report in these callout boxes.
Summary
The main trends that emerged from the survey responses include:
- Overwhelming agreement that the FM symposium is a high quality conference with its distinct character
- A broad range of different opinions regarding the importance and relevance of conference rankings
- Substantial agreement about the importance of diversity (of all kinds) in the PC of FM
- General preference against using metrics such as the h-index to select PC members
- We understand the importance of ranking for a significant part of the FM community
- Despite the recent downranking, we have invested significant effort into improving the standing of FM with ICORE
- We do not endorse the use of metrics like the h-index to evaluate the quality of an event, but this is the base of the vast majority of data requested in an ICORE submission
- We want to continue this discussion with the community, to plan further efforts in this direction
- Our top priority remains to ensure that FM remains a top-quality, yet inclusive venue for the FM community
Respondent Background
The respondents background in terms of work sector, career level, FME membership, and country of residence / work is summarised in Figure 1, Figure 2, Figure 3, and Figure 4.
Rankings and FM
Respondents rated each statement on a 1–5 scale: 1 = Strongly disagree, 3 = Neutral, 5 = Strongly agree. The chart in Figure 5 uses a diverging stacked bar layout centered on the neutral response (3), with disagreement extending left and agreement extending right.
Consistency analysis
Responses to questions about the importance of rankings (Q1, Q4, Q6)2 are consistent, in the sense that respondents who agree that rankings are important (Q1, Q4) also tend to disagree that they should be ignored (Q6): the mean Spearman correlation between Q1, Q4, and reversed Q6 is 0.51.
Responses to questions about the importance of h-index thresholds are also consistent with the perceived importance of rankings: respondents who agree that h-index thresholds should be enforced (Q11, Q12, Q13) are also more likely to agree that FM should try to get a high ranking (Q4). The mean Spearman correlation between these questions is 0.64.
The mean Spearman correlation between questions Q8, Q9, and Q10 about the importance of diversity in the PC and reversed questions Q11, Q12, and Q13 about the importance of h-index thresholds is 0.32, which is weaker than the previous two but still not negligible.
The mean Spearman correlation between question Q5 and reversed questions Q7 is 0.03, which is basically negligible. Therefore, there is no clear relationship between a preference for joining existing rankings (Q5) and for having an FM-specific ranking system (Q7).
Comments about the multiple-choice questions
The vast majority of responses agree that FM is a high-quality conference with a distinct character (Q2, Q3).
We obviously agree too. Ensuring that this quality is maintained, and even improved, remains one of FME’s top priorities.
On the key questions about the importance of rankings (Q1, Q4, Q6) there is a wide spread of opinions with evenly distributed responses and averages around 3.0 (neutral).
This suggests that there is no clear consensus on the importance of rankings, and the community is divided on this issue: for some, rankings are important and should be taken seriously; for others, they are unimportant and should be largely ignored.
The responses mostly agree on the value of diversity in the program committee (PC) of FM, along different dimensions: industry participation (Q8), geographical diversity (Q9), and early-career researchers (Q10). Conversely, the majority of responses are against the idea of using h-index thresholds for selecting PC chairs and members (Q11, Q12, Q13).
This highlights a tension between prioritizing diversity and inclusivity in the PC and strengthening our ICORE submission, which relies heavily on the Google Scholar h-indexes of organizers and authors. The relative consistency between answers to these questions indicates that we should try to navigate this tension and find a reasonable balance. See the comments below for more discussion on this point, and some concrete suggestions for how to address it.
There is a general preference for trying to join ICORE panels (Q5), whereas only a minority of responses are in favor of creating an FM-specific ranking system (Q7).
Regarding the idea of contributing to ICORE’s decision process, we want to acknowledge that we have made a significant effort:
- We have volunteered to join the ICORE panels, but our offer was not accepted without providing any feedback.
- We have submitted an extensive application to ICORE in the last round of evaluations, which is publicly available.
We will continue to try to engage with ICORE and present our case as convincingly as possible. However, we should also point out that the decision process of ICORE is somewhat opaque, which complicates coming up with a successful strategy.
Summary of Open Comments
The following summary of the free-text comments organized by topics and themes is based on an AI-generated3 summary, which we revised, edited, and integrated with our remarks. Overall, 37 respondents (46% of all 80 respondents) provided some additional comments in the free-text question.
Overall Opinions on ICORE/CORE Rankings
As also seen in the multiple-choice questions, there is a wide spread of opinions about the rankings, ranging from negative to positive.
Negative Views: Rankings Are Flawed, Harmful, or Irrelevant
- Rankings are unscientific, opaque, biased, and misaligned with FM values.
- They distort research priorities, damage integrity, and incentivize “gaming the system.”
- Rankings often mismeasure or unfairly penalizespecialized fields, such as formal methods.
- ML and related areas produce high-volume papers, skewing comparisons.
- Rankings create self‑fulfilling downgrades: lower ranking → fewer submissions → lower visibility → further downgrades.
- Criticism of ICORE: “bogus,” “hostage-taking,” “deeply flawed,” “silly,” and “without merit.” Poor transparency: unclear committee recruitment, arbitrary metric usage, opaque evaluation criteria.
Positive Views: Rankings Are Important and Should Not Be Ignored
- Rankings directly impact promotion, funding, travel support, and early‑career opportunities.
- Ignoring rankings is impossible in institutions where they form a primary metric.
- In some countries, rankings play a central role in research evaluation.
- The FM community should take the ICORE ranking seriously because it shapes career outcomes.
ICORE does not really support a clear opt-in/opt-out model where each conference may request to be ranked or decline to be included in the ranking. Instead, they periodically request information from conferences; if this information is not provided in a timely manner,their panels will “assess purely on the basis of the limited data available” which usually results in a major downgrading of the conference (but not removal!) in the rankings.
Mixed Views
- Rankings are flawed but FM should deal with them pragmatically.
- Having a ranking is better than having none.
- FM should redefine excellence beyond metrics but still engage with ranking systems due to institutional realities.
We generally agree with a pragmatic approach: while we do not think rankings are a valid way to measure the quality of research, we also acknowledge that they may have a significant impact on researchers’ careers and opportunities. Hence, we have actively and strategically engaged with the ranking systems, and have striven to position FM as best as possible, but without compromising our values and identity as a community. The next steps in this engagement will be discussed in the upcoming FME meetings.
Descriptions of Consequences
Harm to Early‑Career Researchers
- Early‑career researchers are most impacted by rankings.
- Promotions often require A* publications, disadvantaging FM researchers.
- PhD students face pressure to publish in high-ranked venues even if inappropriate for their work.
Distorted Research Choices
- Researchers feel forced to publish in ranked venues regardless of relevance.
- FM is less attractive to junior researchers due to limited A/A* options.
Institutional / Funding Impacts
- Rankings affect: travel funding, grant competitiveness, departmental evaluations, institutional investment in FM.
- Institutions often perceive low-ranked venues as low-quality regardless of content.
Geographic Differences
- U.S. culture values POPL/PLDI/CAV more; FM is seen as European.
- Some national systems tightly couple funding to ICORE rankings.
These comments confirm that rankings and other metrics have a significant impact on careers. While the magnitude of the impact varies across countries, institutions, and career stages, it is consequential for many researchers.
On the FM Symposium and Community
Some remarks focused on some features of the FM community (and the FM symposium) that may penalize it w.r.t. rankings.
FM Periodicity and Scope
- FM’s 18‑month periodicity harms its ranking prospects: a flagship event must be annual to meet CORE expectations; in contrast, irregular scheduling reduces visibility and violates criteria about maintaining a yearly conference cycle.
As previously decided, starting with FM 2027, FM will switch to an annual schedule and will always take place in the autumn. Among other things, this should also help with the rankings, as it will meet the expectations of a flagship conference.
- Build an ecosystem of workshops and doctoral events around FM.
This is another suggestion that is already being implemented. FM has been growing steadily in recent years, not just in terms of submitted papers but with an increasing number of co-located events. To mention just two examples:
- FM 2024 accepted full-length tutorial papers, which is a distinct category of papers that comply with the same quality standards as regular research papers, but is more focused on pedagogy rather than technical innovation.
- FM 2026 includes a special track on TAP (Tests & Proofs) focused on the intersection of formal verification and software testing.
FM Quality and Evaluation Standards
- The ranking “crisis” is an opportunity to reflect on FM’s identity.
- FM acceptance standards are lower than top‑tier software engineering venues (ICSE/ASE/FSE).
- Papers using simplified or unrealistic examples weaken FM’s impact.
- Enhance: review quality, reproducibility, artefact availability, inter-community collaboration, industrial relevance.
- Reflect on FM’s identity and communicate its strengths externally: FM’s quality is strong but metrics do not capture it.
- FM values: multi‑career‑stage inclusivity, authenticity, industrial impact, methodological depth.
- FM should not reshape itself purely to satisfy external metrics.
We welcome a broader discussion that goes beyond the rankings about the standards of quality and evaluation in FM conferences. Any changes or improvements in this direction would be desirable, but they would not necessarily lead to better ranking.
More to the point of the survey, we note that ICORE uses Google Scholar’s classification of computer science areas into subcategories which does not include Formal Methods as a subcategory, in contrast to other areas such as Theoretical Computer Science and Software Systems. This may explain, at least in part, why FM is disadvantaged in the rankings, when it is compared to conferences in related, but different, areas.
PC Composition and Selection
- Concerns over selecting PC members based on metrics rather than expertise. This may erode FM’s identity and reduce field relevance.
- Suggestion: adopt mixed PC structures (senior + junior), similar to AI conferences. This would increase h‑index presence in senior PC positions while maintaining junior involvement.
We like the idea of a two-tier PC structure, which could allow us to reconcile diversity and involvement of early-career researchers with the need to retain senior researchers in prominent positions.
In addition, a layered PC structure could also help with the increasing number of submissions, which is another issue that has been affecting FM in recent years (a nice problem to have!).
Other Suggestions for Action
Improve FM’s Standing in ICORE
- Submit stronger or more strategic applications to ICORE.
- Let’s try to “game the system” or play strategically within ICORE rules.
Advocate Institutionally
- Campaign to improve recognition of FM by funding bodies.
- Push nationally to reduce the weight of flawed ranking systems.
- Coordinate institutional resistance to rankings.
Create an FM‑Specific Ranking
- Some support creating an FM-focused ranking; others fear it would appear biased, “whacky,” or unfair.
- Concern that new metrics could introduce intra-community conflict.
Boycott or Reject ICORE
- Calls for large-scale boycott or removal of FM from ICORE.
- Argument: rankings harm FM more than they help.
- Caveat: only possible if institutions do not heavily depend on rankings.
We thank all respondents for their participation and for sharing their opinions and suggestions.
While we are fully aware of the negative impact of rankings, we also acknowledge that completely ignoring them is not a realistic option for many researchers. We have done our best to engage with the ranking systems strategically, and to try to position FM as best as possible, but without compromising our values and identity as a community. We are committed to continuing this discussion with the FM community.
We are keen to liase with other computer science communities to try to come up with a coordinated strategy and shared views.
We will also consider other, more far-reaching suggestions, such as campaigning with funding bodies and institutions. However, we are aware that these suggestions are more difficult to implement, and they may not be effective in the short term.
We found the suggestions to improve the FM symposium very valuable. We are delighted to see that several of these suggestions are already being implemented. We are fully committed to continue working on improving the FM symposium in the coming years.
Footnotes
Respondents had the option of revealing their identity, but this report does not include any identifying information.↩︎
We used Microsoft Copilot with a university subscription, which ensures that the data is not used for training, is not human-reviewed, and is stored according to EU data regulations. The LLM was only given the list of free-text answer, without any other (identifying) information from the survey.↩︎