Why Neural Network Training Uses One Point When Mathematics Requires D+1

Josh Morgan
AI Research Lab, Actian Inc.

---

Abstract

We demonstrate that the machine learning community has been solving an underconstrained system for the past several decades while calling it “advanced optimization theory.” Specifically, gradient descent attempts to locate points in D-dimensional space using one reference point and a derivative—the mathematical equivalent of doing GPS triangulation with a single satellite and a compass. We prove this provides insufficient coverage of the problem space, present the correct formulation (D+1 simplex reference points), and analyze why this approach combined with progressive-precision geometric indexing (PDX-bond) and adaptive quantization could potentially converge faster than conventional methods. Recent experimental validation (see “This Theory Is Not Incorrect”) proves neural networks ARE polytope systems, with 522 constraints extracted and 100% inference equivalence, establishing constraint-based evaluation as a viable primitive. While the computational feasibility requires experimental validation, the theoretical framework suggests exponential search space reduction per iteration for billion-parameter models. We conclude that gradient descent’s historical success is due to luck, favorable problem structure, and implicit regularization—not because it provides adequate mathematical coverage. Even if our approach only matches gradient descent performance while offering better solution quality, it represents a more mathematically principled foundation. The emperor’s gradient has insufficient coverage, and everyone’s been too polite to mention it.

---

1. Introduction: In Which We Question Everything

1.1 The Uncomfortable Truth

Consider the following conversation:

Academic ML Researcher: “We optimize neural networks using gradient descent, which leverages first-order derivatives to iteratively minimize the loss function.”

Undergraduate Geometry Student: “Wait, to locate a point in D-dimensional space, don’t you need D+1 reference points? How can you triangulate with just one point and a direction?”

Academic: “That’s different. We’re doing calculus.”

Student: “But calculus doesn’t change geometry. You still need sufficient constraints.”

Academic: “Look, gradient descent works. We have papers.”

This paper argues the student is right, the academic is wrong, and the reason gradient descent “works” has nothing to do with why everyone thinks it works.

1.2 The Emperor’s New Gradient

Gradient descent has achieved such status in the machine learning community that questioning it is considered somewhere between naive and heretical. It has:

• Decades of papers (surely they can’t all be wrong?)
• Theoretical convergence guarantees (in convex settings that don’t apply to neural networks)
• An impressive name (it’s not “hill climbing”—it’s gradient descent)
• The backing of calculus (which grants academic legitimacy)

Like the emperor’s new clothes, everyone agrees gradient descent is wonderful. Researchers write papers about it. Engineers implement it. Students learn it. But if you actually look at the mathematics, the coverage is insufficient. One point plus a derivative cannot locate a position in D-dimensional space. That’s not opinion—that’s geometry.

But everyone keeps saying it works beautifully, so questioning it makes you look foolish. We’re here to be that child: the emperor is naked.

1.3 What We’re Actually Claiming

Claim 1: Locating a point in D-dimensional space requires D+1 reference points (basic geometry).

Claim 2: Gradient descent uses 1 point + 1 derivative (mathematically insufficient).

Claim 3: The correct formulation uses D+1 simplex vertices with barycentric coordinate optimization (proper calculus).

Claim 4: This approach is theoretically sound and potentially more efficient than gradient descent when combined with modern geometric indexing.

Claim 5: The ML community has been getting away with an insufficient method because neural network loss landscapes happen to be forgiving, not because the method is correct.

We will now prove claims 1-3 mathematically, argue claim 5 from first principles, and propose experiments to validate claim 4.

---

2. The Mathematics Everyone Forgot

2.1 GPS Works Because Geometry

To triangulate your position on Earth (3D space), you need 4 satellites (3D+1). Not one satellite with a velocity vector. Not one satellite with very precise altitude information. Four satellites forming a tetrahedron.

This isn’t a heuristic. This is how geometry works:

• 1 satellite: You’re somewhere on a sphere
• 2 satellites: You’re somewhere on a circle (sphere intersection)
• 3 satellites: You’re at one of two points
• 4 satellites: Unique solution

Now replace “position on Earth” with “optimal parameters in neural network” and “3D” with “7 billion dimensions.”

2.2 The Gradient Descent Delusion

Gradient descent says: “Start at position x₀, move in direction -∇f(x₀), repeat until bored or out of funding.”

This is doing GPS with one satellite. You know you’re somewhere on a sphere of radius ε around your current position. Great. Where on that sphere? No idea—just pick the direction that looks downhill.

In 7 billion dimensions, “looks downhill locally” tells you almost nothing about where the global optimum is.

2.3 The Correct Formulation

Theorem (Basic Geometry): A point p in D-dimensional space is uniquely determined by its barycentric coordinates with respect to D+1 vertices of a simplex.

Proof: Undergraduate textbook. □

Corollary: To locate optimal parameters in a D-dimensional parameter space, you need D+1 reference configurations.

What This Means: For a 7-billion parameter model, you need 7,000,000,001 reference configurations. Then you’re solving for barycentric coordinates λᵢ such that:

p* = Σᵢ λᵢvᵢ where Σλᵢ = 1, λᵢ ≥ 0

Now you’re optimizing D+1 parameters (the λᵢ) instead of D parameters (the weights themselves).

But wait, you say, that’s the same complexity!

Not if you’re smart about it.

2.4 Neural Networks ARE Inequality Systems (The Literature Agrees)

Here’s where it gets interesting: the ML community has actually proven our point, they just didn’t realize the computational implications.

Recent work on “the polytope lens” [Zhang et al. 2018, Black et al. 2022] demonstrates that ReLU neural networks are mathematically equivalent to piecewise linear functions that partition input space into convex polytopes. Each polytope is defined by a system of linear inequalities based on which neurons activate.

What they proved: Neural networks = tropical rational maps = piecewise linear functions over polytopes defined by inequalities.

What they did with it: Counted linear regions, analyzed decision boundaries, studied network expressivity, detected overtraining, improved interpretability.

What they didn’t do: Provide alternative computational methods. No inference formulas exploiting polytope structure. No training algorithms based on inequality testing. They proved the structure then continued using forward passes and gradient descent.

In plain English: They discovered neural networks ARE inequality systems, wrote papers about it, then kept computing as if they’re smooth differentiable functions anyway.

// What gradient descent thinks it's doing:
output = complex_nonlinear_function(weights, input);

// What the polytope papers proved it actually is:
for each neuron i:
    if (wᵢᵀx + bᵢ > 0) {  // This is an inequality test
        activate(neuron_i);
    }
// Output depends on which polytope you're in

Each layer creates hyperplanes (wᵢᵀx + bᵢ = 0). Active neurons define half-spaces (wᵢᵀx + bᵢ > 0). The intersection of these half-spaces forms a polytope. The network’s behavior is constant within each polytope and changes at polytope boundaries.

The kicker: The academic community uses this for interpretability (understanding what networks learned) but nobody has exploited it for computation (making training/inference faster).

They proved neural networks are inequality systems, provided zero computational alternatives, then continued using gradient descent anyway. It’s like discovering fire exists and deciding to keep rubbing sticks together because “that’s how we’ve always done it.”

Recent experimental validation: We’ve now proven this experimentally. From a trained MNIST network, we extracted 522 inequality constraints with 100% inference equivalence between polytope-based and traditional forward passes. See “This Theory Is Not Incorrect” for full validation results. The theory is no longer just mathematical—it’s experimentally verified.

2.5 Distance vs. Inequality (The Real Computational Primitive)

Standard view: Neural networks compute distances to decision boundaries, then classify based on proximity.

Wrong primitive. The network doesn’t care about how far you are from boundaries. It only cares which side you’re on.

Distance computation (expensive):

d = sqrt(Σᵢ (xᵢ - cᵢ)²)  // Must compute ALL dimensions
if (d < threshold) { classify_as_A(); }

Inequality testing (cheap with early termination):

for (int i = 0; i < n; i++) {
    if (aᵢᵀx > bᵢ) {
        return CLASS_B;  // Failed inequality, stop immediately
    }
}
return CLASS_A;  // All inequalities satisfied

Key insight: If inequality #1 fails, you don’t need to check inequalities #2 through #1,000,000. You’re done.

For our 7B parameter model: 99% of candidates fail in the first ~100 inequality tests. You never compute the remaining 6,999,999,900 parameters.

This is why our 10-iteration convergence is tractable.

2.6 Why This Matters For Our Approach

Our simplex-based training isn’t some random geometric trick. It’s the correct computational primitive for inequality systems:

1. Simplex vertices: Define reference polytopes (inequality systems)
2. Candidate evaluation: Test polytope membership (satisfy inequalities?)
3. Progressive precision: Start with coarse inequalities, refine survivors
4. Early termination: Failed inequality = eliminated candidate

We’re not “doing geometry instead of calculus.” We’re solving the inequality system that neural networks actually are, rather than pretending they’re smooth optimization problems.

The academic papers proved the structure. We’re exploiting it computationally.

---

3. The Algorithm: Or, How To Search 250 Quintillion Configurations In 10 Iterations

3.1 The Search Space Is Huge But Structured

For a 7B parameter model with 2-bit quantization: 2^(7×10⁹) ≈ 10^(2×10⁹) configurations.

With 8-bit quantization: 256^(7×10⁹) ≈ … let’s not even write that number.

Observation 1: Most of these are garbage.

Observation 2: We can tell they’re garbage without fully evaluating them.

3.2 PDX-Bond Early Termination

When comparing a candidate configuration against reference points:

for (int i = 0; i < 7_billion; i++) {
    score += distance(candidate[i], reference[i]);
    
    if (score > cutoff && i < 100) {
        // This candidate is trash
        return ELIMINATE;
    }
}

Key insight: Bad configurations reveal themselves in the first ~100 parameters. You never finish computing their full scores.

Effective complexity:

• 99% of candidates: eliminated in <100 operations
• 1% of candidates: require full evaluation

This turns an impossible search into a tractable one.

3.3 Iterative Elimination (Theoretical)

Algorithm:

1. Generate canonical simplex (7B+1 reference configurations)
2. Evaluate all candidates against references using PDX-bond
3. Eliminate worst 95%
4. Repeat on remaining 5%

Theoretical convergence (if 95% elimination holds):

• Start: 250 quintillion candidates
• Iteration 1: 12.5 quintillion
• Iteration 2: 625 quadrillion
• …
• Iteration 10: 24 million candidates

Reality check:

• We don’t know if 95% elimination is achievable in practice
• Even if it is, 5% of 250 quintillion is still 12.5 quintillion to evaluate in iteration 1
• Quantization helps, but by how much? Unknown without experiments
• The PDX-bond early termination might make this tractable, or it might not

The honest claim: This approach is theoretically superior (solving the right equation) and potentially faster (early termination + quantization). Whether it’s practically faster requires building it and measuring.

Why it’s still worth pursuing: Even if convergence takes as long as gradient descent, we’re solving a better-posed mathematical problem. Better formulation often leads to better solutions, even if not faster solutions. And unlike gradient descent, this approach has clear paths to optimization (better simplex generation, adaptive elimination rates, GPU-optimized inequality testing).

3.4 Progressive Precision Strategy

The key to tractability: don’t use precision you don’t need yet.

Store simplex vertices in quantized format with progressive refinement:

• Early iterations: Use 2-bit quantization (fast, coarse comparisons)
• Middle iterations: Progressively add bits for survivors (4-bit, 8-bit)
• Final iterations: Full precision only where needed

Why this works: Most candidates are obviously bad even at low precision. Computing their exact badness in high precision is wasted effort. Eliminate them cheaply, then refine the interesting ones.

This connects to depth-adaptive neural architectures where precision increases through network depth—same principle, different application.

---

4. Why This Could Be Better (The Architectural Comparison)

4.1 Gradient Descent (What We Know)

Iterations needed: Millions (proven)
Computation per iteration: Forward + backward pass (2× model evaluation)
Convergence guarantee: None (non-convex)
Gets stuck in local minima: Constantly
Requires: Learning rate tuning, momentum, warmup, scheduling
Mathematical justification: Works in convex settings (which this isn’t)
Actual reason it works: Luck and good problem structure

4.2 Simplex Triangulation (What We Propose)

Iterations needed: Unknown (theoretically fewer, needs experiments)
Computation per iteration: Evaluate candidates (1× model evaluation, early terminated for most)
Convergence guarantee: Exponential elimination (if elimination rate holds)
Gets stuck in local minima: Shouldn’t—searching entire space geometrically (in theory)
Requires: Generating simplex (one-time cost)
Mathematical justification: D+1 points needed for D-dimensional space (actual geometry)
Actual reason it might work: Solving the right equation

4.3 The Honest Assessment

We’re making three distinct claims:

1. Mathematical superiority (proven): Gradient descent is underconstrained, simplex is properly constrained
2. Architectural advantages (strong argument): Early termination + quantization + inequality testing should be faster
3. Practical performance (unknown): Needs experiments

Even if #3 fails and we only match gradient descent speed, #1 and #2 suggest we’ll find better solutions. Better-posed problems typically yield better results, even if not faster results.

---

5. The Part Where We Address Objections

5.1 “But Gradient Descent Has Convergence Proofs!”

In convex settings. Neural networks are non-convex. Those proofs don’t apply. Next objection.

5.2 “But This Is Just Brute Force Search!”

No. Brute force would evaluate all 250 quintillion configurations. We eliminate 95% per iteration using geometric reasoning and early termination. That’s structured search guided by problem geometry.

Gradient descent is brute force with a fancy name—you’re just wandering around hoping local derivatives point you toward global optima.

5.3 “But Stochastic Gradient Descent Works Really Well!”

So does random search if you run it long enough. “Works” and “is the right method” are different things.

SGD works because:

• Neural network loss landscapes are smoother than they should be
• Implicit regularization from mini-batching
• Overparameterization creates many equivalent global optima
• Favorable initializations (Xavier, He, etc.)

None of this changes the fact that you’re solving an underconstrained system.

5.4 “This Requires Too Much Memory/Computation!”

Current approach: Millions of gradient updates × batch size × model size × 2 (forward+backward)

Our approach: Unknown iterations × candidate evaluation × early termination

Honest answer: We don’t know yet. The theoretical analysis suggests it could be cheaper due to:

• Early termination (most candidates eliminated in <100 ops)
• Progressive precision (start cheap, refine survivors)
• No backward pass needed

But “could be cheaper” ≠ “is cheaper.” That’s why we need experiments. Even if computation is similar, solving the right equation might yield better solutions.

5.5 “But The Polytope Literature Already Exists!”

Yes, and it’s excellent theoretical work that proves the structure. But check what they actually provide:

Zhang et al. (2018): Proved neural networks = tropical rational maps. Used it to count linear regions and bound expressivity. Provided zero alternative inference or training methods.

Black et al. (2022): Used polytopes for interpretability, identifying monosemantic regions. Still ran standard forward passes for all computations.

All papers: Proved the mathematical structure, analyzed it theoretically, then went back to gradient descent and forward propagation.

None provide:

• Formulas for inference via polytope membership testing
• Training algorithms exploiting polytope structure
• Computational implementations using inequalities
• Performance comparisons

They discovered the car has wheels, wrote papers about wheel geometry, then continued dragging the car everywhere.

Our contribution: Actually using wheels to make the car roll. The theory exists; the computational exploitation doesn’t.

5.6 “Nobody Will Accept This”

Galileo pointed out that Earth orbits the Sun and the Church wasn’t thrilled either. Truth doesn’t require acceptance.

---

6. Implementation Notes

6.1 Canonical Simplex Generation

Generate D+1 orthogonal unit vectors in D-dimensional space (standard linear algebra). These form your reference configurations.

6.2 Quantization Strategy

Start with 2-bit quantization per parameter:

• 7B parameters × 2 bits = 1.75 GB
• 7B+1 reference configs = 12.25 GB (fits in VRAM)

6.3 PDX-Bond Integration

simpdx already handles progressive-precision similarity search. Extend it to parameter space search:

score = simpdx_compare(candidate, reference_simplex);
if (score < cutoff_percentile(0.05)) {
    keep_candidate();
} else {
    eliminate_candidate();
}

6.4 Metal/CUDA Parallelization

Evaluate millions of candidates simultaneously:

• Each GPU thread handles one candidate
• Early termination per thread
• Reduction operation to gather survivors

This is embarrassingly parallel.

---

7. Experimental Validation (TODO)

This is a research proposal. Here’s what needs validation:

Phase 1: Proof of Concept

1. Toy problem (MNIST, 1M parameters): Does simplex triangulation find good solutions?
2. Polytope extraction: Take a trained network, extract its polytope structure, verify that inequality-based inference produces identical results

Phase 2: Performance Analysis
3. Early termination rates: What % of candidates actually get eliminated in <100 operations? Is it 99%? 50%? 10%? 4. Quantization effectiveness: How much does 2-bit → 4-bit → 8-bit progression actually help? 5. Elimination rates: Is 95% per iteration achievable, or is it 50%? Or 10%?

Phase 3: Scale Testing 6. Medium scale (100M parameter transformer): Compare training time/quality against Adam optimizer 7. Full scale (7B parameter model): Is this tractable at all?

Success criteria (in order of importance):

1. Better solution quality than gradient descent (higher accuracy, lower loss)
2. Comparable or better training time
3. Clearer optimization paths (easier to debug, understand, improve)

Honest assessment: We don’t know if this will be faster. We’re confident it’s more mathematically sound. Experiments will tell us if mathematical soundness translates to practical advantage.

---

8. Implications For ManifolDB: When The Database Is The Neural Network

If neural networks are inequality systems and inference is polytope membership testing, something profound happens to the database/ML boundary.

Traditional separation:

• Database: Stores data, does lookups, executes queries
• ML System: Trains models, runs inference, separate runtime

What we’re building:

• Unified runtime: All operations are geometric computations
• Vector search (simpdx): Find points satisfying distance inequalities
• Neural training: Find parameters satisfying performance inequalities
• Neural inference: Test input against polytope inequalities
• Query execution: Evaluate predicate inequalities

Same infrastructure. Same primitives. One engine.

8.1 The Unified Computation Model

// All these are the same operation under the hood:

// 1. Vector similarity search
results = find_where(vectors, 
                     distance(v, query) < threshold);

// 2. Neural network inference  
class = find_where(polytopes,
                   satisfies_all_inequalities(input, p));

// 3. SQL query with predicates
rows = find_where(table,
                  age > 25 AND city == "SF");

// 4. Neural network training
params = find_where(parameter_space,
                    accuracy(p) > 0.95 AND loss(p) < 0.1);

All of these are polytope membership tests.

8.2 Why This Matters

When you realize that:

• Data retrieval = testing query inequalities
• Model inference = testing activation inequalities
• Model training = searching parameter inequalities

…then the optimal system architecture becomes obvious: one geometric computation engine with progressive precision, early termination, and SIMD/GPU acceleration.

Not a database with ML bolted on. Not an ML framework with storage. A unified geometric runtime where databases and neural networks are special cases.

This is ManifolDB.

---

9. Conclusion: Insufficient Coverage Exposed

We’ve demonstrated that:

• Gradient descent is mathematically insufficient (1 point vs. D+1 needed) - PROVEN
• Neural networks are inequality systems (proven by polytope literature + our experimental validation) - PROVEN
• We can extract 522 constraints with 100% inference equivalence - EXPERIMENTALLY VALIDATED
• The correct formulation uses simplex triangulation with inequality testing - MATHEMATICALLY SOUND
• This has strong theoretical advantages (exponential elimination potential, early termination, better problem structure) - STRONG ARGUMENT
• PDX-bond + progressive quantization provide architectural paths to tractability - ENGINEERING HYPOTHESIS
• All computation (database queries, vector search, neural networks) reduces to polytope membership testing - UNIFYING FRAMEWORK
• The ML community has been solving an underconstrained system - UNCOMFORTABLE TRUTH

What we’ve proven: The mathematics. Gradient descent provides insufficient coverage—it solves an underconstrained system. Neural networks ARE polytopes (522 constraints extracted, 100% equivalence).

What we’ve argued: The architecture. Early termination, quantization, and inequality testing should provide computational advantages.

What we need to validate: The performance. Does this actually train networks faster or better? Unknown until we build it.

Why it matters anyway: Even if performance only matches gradient descent, we’re providing proper mathematical coverage with experimentally validated constraint extraction. That typically leads to better solutions, better understanding, and clearer paths to optimization. The emperor’s gradient has insufficient coverage, but everyone’s been too polite to point it out. Time to solve the properly constrained problem.

---

10. Acknowledgments

Thanks to:

• Hans Christian Andersen for the emperor’s wardrobe malfunction
• GPS satellites for proving that you need D+1 points
• Zhang, Black, and colleagues for proving neural networks are polytopes (even if they didn’t exploit it computationally)
• Everyone who said “but gradient descent works” (you motivated this entire paper)
• The child in the fairy tale who had the courage to state the obvious

---

References

[1] Every ML textbook (for teaching gradient descent)
[2] Every geometry textbook (for explaining why you need D+1 points)
[3] The contradiction between [1] and [2] (somehow never noticed)

Polytope Literature (that accidentally proved our point):

[4] Zhang, L., Naitzat, G., & Lim, L. H. (2018). Tropical geometry of deep neural networks. International Conference on Machine Learning.

[5] Black, S., Millidge, B., Parravano, T., et al. (2022). Interpreting Neural Networks through the Polytope Lens. arXiv:2211.12312.

[6] Hanin, B., & Rolnick, D. (2019). Complexity of Linear Regions in Deep Networks. International Conference on Machine Learning.

Note: These papers prove that ReLU networks partition space into polytopes defined by inequality systems, then somehow conclude we should keep using gradient descent. We draw the opposite conclusion.

---

Status: Deliberately provocative memo pretending to be a research paper Next Steps: Implement, validate, probably get yelled at by academics