Separate core specific Request Writers from node specific "built in" ones. Move core specific to using ImplicitPlugins.json. #4073
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| # See https://github.com/apache/solr/blob/main/dev-docs/changelog.adoc | ||
| title: Introduce minimal ADMIN_RESPONSE_WRITERS for admin/container-level requests. Admin requests now use a minimal set of 6 response writers (javabin, json, xml, prometheus, openmetrics, standard) instead of the full 14-writer DEFAULT_RESPONSE_WRITERS map. Core-specific requests continue to use ImplicitPlugins.json with ConfigOverlay support. | ||
| type: changed | ||
|
epugh marked this conversation as resolved.
Outdated
|
||
| authors: | ||
| - name: Eric Pugh | ||
| links: | ||
| - name: GitHub Discussion | ||
| url: https://github.com/apache/solr/discussions | ||
| notes: | | ||
| Admin/container-level requests (e.g., /admin/cores, /admin/collections) have no associated SolrCore | ||
| and cannot use the core's response writer registry loaded from ImplicitPlugins.json. Previously, | ||
| these requests used DEFAULT_RESPONSE_WRITERS (14 entries). Now they use ADMIN_RESPONSE_WRITERS (6 entries) | ||
| containing only the formats actually needed by admin APIs: | ||
| - javabin (SolrJ clients) | ||
| - json (Admin UI) | ||
| - xml (backward compatibility) | ||
| - prometheus/openmetrics (metrics endpoint) | ||
| - standard (alias for json) | ||
|
|
||
| This provides clearer separation of concerns and better documentation of requirements. Core-specific | ||
| requests are unaffected and continue to use the full ImplicitPlugins.json system with ConfigOverlay support. | ||
| DEFAULT_RESPONSE_WRITERS remains available but is now deprecated for internal use. | ||
|
There was a problem hiding this comment. The changelog entry refers to an "ADMIN_RESPONSE_WRITERS" name and says "DEFAULT_RESPONSE_WRITERS remains available". In this PR, DEFAULT_RESPONSE_WRITERS is removed from SolrCore and replaced by BuiltInResponseWriterRegistry (and a deprecated SolrCore.getAdminResponseWriter method). Please update this changelog text to match the actual public/internal API surface introduced by the PR so release notes aren’t misleading. |
||
| Original file line number | Diff line number | Diff line change | ||||||
|---|---|---|---|---|---|---|---|---|
|
|
@@ -17,8 +17,6 @@ | |||||||
| package org.apache.solr.core; | ||||||||
|
|
||||||||
| import static org.apache.solr.common.params.CommonParams.PATH; | ||||||||
| import static org.apache.solr.metrics.SolrCoreMetricManager.COLLECTION_ATTR; | ||||||||
| import static org.apache.solr.metrics.SolrCoreMetricManager.CORE_ATTR; | ||||||||
| import static org.apache.solr.metrics.SolrCoreMetricManager.REPLICA_TYPE_ATTR; | ||||||||
|
|
@@ -129,19 +127,9 @@ | |||||||
| import org.apache.solr.request.SolrQueryRequest; | ||||||||
| import org.apache.solr.request.SolrRequestHandler; | ||||||||
| import org.apache.solr.request.SolrRequestInfo; | ||||||||
| import org.apache.solr.response.JavaBinResponseWriter; | ||||||||
| import org.apache.solr.response.QueryResponseWriter; | ||||||||
| import org.apache.solr.response.SolrQueryResponse; | ||||||||
| import org.apache.solr.response.transform.TransformerFactory; | ||||||||
| import org.apache.solr.rest.ManagedResourceStorage; | ||||||||
| import org.apache.solr.rest.ManagedResourceStorage.StorageIO; | ||||||||
|
|
@@ -3088,26 +3076,6 @@ public PluginBag<QueryResponseWriter> getResponseWriters() { | |||||||
|
|
||||||||
| private final PluginBag<QueryResponseWriter> responseWriters = | ||||||||
| new PluginBag<>(QueryResponseWriter.class, this); | ||||||||
|
|
||||||||
| private static JavaBinResponseWriter getFileStreamWriter() { | ||||||||
| return new JavaBinResponseWriter() { | ||||||||
|
|
@@ -3148,11 +3116,49 @@ default String getContentType() { | |||||||
| } | ||||||||
|
|
||||||||
| /** | ||||||||
| * Gets a response writer suitable for admin/container-level requests. | ||||||||
|
epugh marked this conversation as resolved.
Outdated
|
||||||||
| * | ||||||||
| * @param writerName the writer name, or null for default | ||||||||
| * @return the response writer, never null | ||||||||
| * @deprecated Use {@link | ||||||||
| * org.apache.solr.response.BuiltInResponseWriterRegistry#getWriter(String)} instead. | ||||||||
| */ | ||||||||
| @Deprecated | ||||||||
| public static QueryResponseWriter getAdminResponseWriter(String writerName) { | ||||||||
| return org.apache.solr.response.BuiltInResponseWriterRegistry.getWriter(writerName); | ||||||||
| } | ||||||||
|
|
||||||||
| /** | ||||||||
| * Initializes query response writers. Response writers from {@code ImplicitPlugins.json} may also | ||||||||
| * be configured. | ||||||||
| */ | ||||||||
| private void initWriters() { | ||||||||
| // Build default writers map from implicit plugins | ||||||||
| Map<String, QueryResponseWriter> defaultWriters = new HashMap<>(); | ||||||||
|
|
||||||||
| // Load writers from ImplicitPlugins.json | ||||||||
| List<PluginInfo> implicitWriters = getImplicitResponseWriters(); | ||||||||
| for (PluginInfo info : implicitWriters) { | ||||||||
| try { | ||||||||
| QueryResponseWriter writer = | ||||||||
| createInstance( | ||||||||
| info.className, | ||||||||
| QueryResponseWriter.class, | ||||||||
| "queryResponseWriter", | ||||||||
| null, | ||||||||
|
There was a problem hiding this comment. SolrCore.initWriters() passes
Suggested change
Contributor
Author
There was a problem hiding this comment. This is interesting, and I think a separte issue. None of the QueryResponseWriter's take a core. RawResponseWriter looks up a core from a specific request, but doesn't take it directly. There are. a few calls to the createInstance and they mostly pass a
Contributor
Author
There was a problem hiding this comment. I would love to have a version of
Contributor
There was a problem hiding this comment. is this a change -- were response writers receiving the core and now, not?
Contributor
Author
There was a problem hiding this comment. this was NOT a change. Only in one instance (that of |
||||||||
| getResourceLoader()); | ||||||||
|
There was a problem hiding this comment. SolrCore.initWriters() instantiates implicit response writers via createInstance(...) but does not run plugin initialization (init args / PluginInfoInitialized / NamedListInitializedPlugin) like other Solr plugins do (see SolrCore.createInitInstance / SolrCore.initPlugin). This can break response writers that expect initialization from PluginInfo. After instantiating, call SolrCore.initPlugin(info, writer) (or use createInitInstance with a PluginInfo) before registering it.
Suggested change
Contributor
Author
There was a problem hiding this comment. this is probably a "good thing" yet none of our QueryResponseWriters need it, so we don't have it. If this plugin creation was centralized more, then it might happen.
Contributor
There was a problem hiding this comment. Great feedback from Copilot.
Contributor
There was a problem hiding this comment. I suspect the VelocityResponseWriter needs a SolrResourceLoader (from a core). Any way, we need not concern ourselves with that. |
||||||||
| defaultWriters.put(info.name, writer); | ||||||||
| } catch (Exception e) { | ||||||||
| log.warn("Failed to load implicit response writer: {}", info.name, e); | ||||||||
| } | ||||||||
| } | ||||||||
|
|
||||||||
| // Add special filestream writer (custom implementation) | ||||||||
| defaultWriters.put(ReplicationAPIBase.FILE_STREAM, getFileStreamWriter()); | ||||||||
|
Contributor
Author
There was a problem hiding this comment. FYI, I don't know why this is "special". I looked, and I think we could just create a class called
Contributor
There was a problem hiding this comment. Glad you looked into this. Seems in-scope to include.
Contributor
Author
There was a problem hiding this comment. great... Going to roll that into this PR. |
||||||||
|
|
||||||||
| // Initialize with the built defaults | ||||||||
| responseWriters.init(defaultWriters, this); | ||||||||
|
|
||||||||
| // configure the default response writer; this one should never be null | ||||||||
| if (responseWriters.getDefault() == null) responseWriters.setDefault("standard"); | ||||||||
| } | ||||||||
|
Comment on lines
+3127
to
3137
There was a problem hiding this comment. SolrCore.initWriters() swallows exceptions when loading implicit response writers (logs warn and continues). If the "standard" writer (or any required default) fails to load, responseWriters may end up with no default (PluginBag#setDefault is a no-op when the name is missing), violating the “Never null” contract of getQueryResponseWriter(..) and potentially causing NPEs later. Consider failing fast when required writers can’t be loaded, or explicitly ensure a default writer is present (e.g., fall back to a built-in JSON writer) before continuing startup. |
||||||||
|
|
@@ -3614,32 +3620,49 @@ public void cleanupOldIndexDirectories(boolean reload) { | |||||||
| } | ||||||||
| } | ||||||||
|
|
||||||||
| private static final class ImplicitPluginsHolder { | ||||||||
| private ImplicitPluginsHolder() {} | ||||||||
|
|
||||||||
| private static final Map<String, List<PluginInfo>> ALL_IMPLICIT_PLUGINS; | ||||||||
|
|
||||||||
| static { | ||||||||
| @SuppressWarnings("unchecked") | ||||||||
| Map<String, ?> implicitPluginsInfo = | ||||||||
| (Map<String, ?>) | ||||||||
| Utils.fromJSONResource(SolrCore.class.getClassLoader(), "ImplicitPlugins.json"); | ||||||||
|
|
||||||||
| Map<String, List<PluginInfo>> plugins = new HashMap<>(); | ||||||||
|
|
||||||||
| // Load all plugin types from the JSON | ||||||||
| for (Map.Entry<String, ?> entry : implicitPluginsInfo.entrySet()) { | ||||||||
| String pluginType = entry.getKey(); | ||||||||
| @SuppressWarnings("unchecked") | ||||||||
| Map<String, Map<String, Object>> pluginConfigs = | ||||||||
| (Map<String, Map<String, Object>>) entry.getValue(); | ||||||||
|
|
||||||||
| List<PluginInfo> pluginInfos = new ArrayList<>(pluginConfigs.size()); | ||||||||
| for (Map.Entry<String, Map<String, Object>> plugin : pluginConfigs.entrySet()) { | ||||||||
| Map<String, Object> info = plugin.getValue(); | ||||||||
| info.put(CommonParams.NAME, plugin.getKey()); | ||||||||
| pluginInfos.add(new PluginInfo(pluginType, info)); | ||||||||
| } | ||||||||
| plugins.put(pluginType, Collections.unmodifiableList(pluginInfos)); | ||||||||
| } | ||||||||
|
|
||||||||
| ALL_IMPLICIT_PLUGINS = Collections.unmodifiableMap(plugins); | ||||||||
| } | ||||||||
|
|
||||||||
| public static List<PluginInfo> getImplicitPlugins(String type) { | ||||||||
| return ALL_IMPLICIT_PLUGINS.getOrDefault(type, Collections.emptyList()); | ||||||||
| } | ||||||||
| } | ||||||||
|
|
||||||||
| public List<PluginInfo> getImplicitHandlers() { | ||||||||
| return ImplicitPluginsHolder.getImplicitPlugins(SolrRequestHandler.TYPE); | ||||||||
| } | ||||||||
|
|
||||||||
| public List<PluginInfo> getImplicitResponseWriters() { | ||||||||
| return ImplicitPluginsHolder.getImplicitPlugins("queryResponseWriter"); | ||||||||
| } | ||||||||
|
|
||||||||
| public CancellableQueryTracker getCancellableQueryTracker() { | ||||||||
|
|
||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,115 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one or more | ||
| * contributor license agreements. See the NOTICE file distributed with | ||
| * this work for additional information regarding copyright ownership. | ||
| * The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| * (the "License"); you may not use this file except in compliance with | ||
| * the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
| package org.apache.solr.response; | ||
|
|
||
| import static org.apache.solr.handler.admin.MetricsHandler.OPEN_METRICS_WT; | ||
| import static org.apache.solr.handler.admin.MetricsHandler.PROMETHEUS_METRICS_WT; | ||
|
|
||
| import java.util.Collections; | ||
| import java.util.HashMap; | ||
| import java.util.Map; | ||
| import org.apache.solr.common.params.CommonParams; | ||
|
|
||
| /** | ||
| * Registry for built-in response writers in Solr. | ||
| * | ||
| * <p>Manages a minimal set of essential response writers that are always available, regardless of | ||
| * core configuration or ImplicitPlugins.json settings. These writers are primarily used by | ||
| * admin/container-level requests that have no associated SolrCore. | ||
| * | ||
| * <p>Built-in writers include essential formats needed by admin APIs and core functionality: | ||
| * javabin, json, xml, prometheus, and openmetrics. | ||
| * | ||
| * <p>For core-specific requests that need access to the full set of response writers (including | ||
| * csv, geojson, graphml, smile, etc.), use the SolrCore's response writer registry which is loaded | ||
| * from ImplicitPlugins.json and supports ConfigOverlay customizations. | ||
| * | ||
| * @since 11.0 | ||
| */ | ||
| public class BuiltInResponseWriterRegistry { | ||
|
|
||
| /** | ||
| * Built-in response writers that are always available. | ||
| * | ||
| * <p>Contains only essential formats needed by admin APIs and core functionality: | ||
| * | ||
| * <ul> | ||
| * <li><b>javabin</b> - Binary format, efficient for SolrJ clients | ||
| * <li><b>json/standard</b> - JSON format, default for most requests | ||
| * <li><b>xml</b> - XML format, provides backward compatibility | ||
| * <li><b>prometheus/openmetrics</b> - Required by metrics endpoints | ||
| * </ul> | ||
| */ | ||
| private static final Map<String, QueryResponseWriter> BUILTIN_WRITERS; | ||
|
|
||
| static { | ||
| // Initialize built-in writers that are always available | ||
| Map<String, QueryResponseWriter> builtinWriters = new HashMap<>(6, 1); | ||
| builtinWriters.put(CommonParams.JAVABIN, new JavaBinResponseWriter()); | ||
| builtinWriters.put(CommonParams.JSON, new JacksonJsonWriter()); | ||
| builtinWriters.put("standard", builtinWriters.get(CommonParams.JSON)); // Alias for JSON | ||
| builtinWriters.put("xml", new XMLResponseWriter()); | ||
| builtinWriters.put(PROMETHEUS_METRICS_WT, new PrometheusResponseWriter()); | ||
| builtinWriters.put(OPEN_METRICS_WT, new PrometheusResponseWriter()); | ||
| BUILTIN_WRITERS = Collections.unmodifiableMap(builtinWriters); | ||
| } | ||
|
|
||
| /** | ||
| * Gets a built-in response writer. | ||
| * | ||
| * <p>Built-in writers are always available and provide essential formats needed by admin APIs and | ||
| * core functionality. They do not depend on core configuration or ImplicitPlugins.json settings. | ||
| * | ||
| * <p>If the requested writer is not available, returns the "standard" (JSON) writer as a | ||
| * fallback. This ensures requests always get a valid response format. | ||
| * | ||
| * @param writerName the writer name (e.g., "json", "xml", "javabin"), or null for default | ||
| * @return the response writer, never null (returns "standard"/JSON if not found) | ||
| */ | ||
| public static QueryResponseWriter getWriter(String writerName) { | ||
| // carrying over this "standard" thing from original code, but do we want this? null/blank | ||
| // means standard? | ||
| // feels like null or blank should throw an exception. And a method should be added that is | ||
| // getWriter() that | ||
| // returns the json guy. I hate passing around nulls and ""... | ||
| if (writerName == null || writerName.isEmpty()) { | ||
| return BUILTIN_WRITERS.get("standard"); | ||
| } | ||
| return BUILTIN_WRITERS.getOrDefault(writerName, BUILTIN_WRITERS.get("standard")); | ||
| } | ||
|
|
||
| /** | ||
| * Returns the set of built-in writer names. | ||
| * | ||
| * <p>This is useful for debugging, testing, and API documentation. The returned set is immutable. | ||
| * | ||
| * @return unmodifiable set of built-in writer names | ||
| */ | ||
| static java.util.Set<String> getWriterNames() { | ||
| return BUILTIN_WRITERS.keySet(); | ||
| } | ||
|
|
||
| /** | ||
| * Checks if a writer is available as a built-in writer. | ||
| * | ||
| * @param writerName the writer name to check | ||
| * @return true if the writer is available as a built-in writer | ||
| */ | ||
| static boolean hasWriter(String writerName) { | ||
| return BUILTIN_WRITERS.containsKey(writerName); | ||
| } | ||
| } |
| Original file line number | Diff line number | Diff line change | ||
|---|---|---|---|---|
|
|
@@ -85,6 +85,7 @@ | |||
| import org.apache.solr.request.SolrQueryRequestBase; | ||||
| import org.apache.solr.request.SolrRequestHandler; | ||||
| import org.apache.solr.request.SolrRequestInfo; | ||||
| import org.apache.solr.response.BuiltInResponseWriterRegistry; | ||||
| import org.apache.solr.response.QueryResponseWriter; | ||||
| import org.apache.solr.response.SolrQueryResponse; | ||||
| import org.apache.solr.security.AuditEvent; | ||||
|
|
@@ -735,8 +736,9 @@ protected void logAndFlushAdminRequest(SolrQueryResponse solrResp) throws IOExce | |||
| solrResp.getToLogAsString("[admin]")); | ||||
| } | ||||
| } | ||||
| // Admin requests have no core, use built-in writers | ||||
|
Contributor
Author
There was a problem hiding this comment. Should we be calling these "Cluster requests" or "Builtin requests"? Admin isn't really a very precise term since we would call "Add a shard" a "admin request"....? Or make sure that when we say "Admin" we mean "Solr Cluster Administration"??? |
||||
| QueryResponseWriter respWriter = | ||||
| BuiltInResponseWriterRegistry.getWriter(solrReq.getParams().get(CommonParams.WT)); | ||||
| if (respWriter == null) respWriter = getResponseWriter(); | ||||
|
There was a problem hiding this comment. HttpSolrCall.logAndFlushAdminRequest() still checks
Suggested change
Contributor
Author
There was a problem hiding this comment. this is a fair comment. It's that werid "standard" thing. We should either solve the null or delete this. |
||||
| writeResponse(solrResp, respWriter, Method.getMethod(req.getMethod())); | ||||
| if (shouldAudit()) { | ||||
|
|
||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I need to redo this... Claude generated (and committed! bad claude!) this...